6.style_GAN

2020-02-29

PyTorch入门与实战第六课

图片风格迁移 Neural Style Transfer

A Neural Algorithm of Artistic Style
本文介绍了Neural Style Transfor模型

%matplotlib inline

from __future__ import division
from torchvision import models
from torchvision import transforms
from PIL import Image
import argparse
import torch
import torchvision
import torch.nn as nn
import numpy as np

import matplotlib.pyplot as plt

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

def load_image(image_path, transform=None, max_size=None, shape=None):
    image = Image.open(image_path)
    if max_size:
        scale = max_size / max(image.size)
        size= np.array(image.size) * scale
        image = image.resize(size.astype(int), Image.ANTIALIAS)
         
    if shape:
        image = image.resize(shape, Image.LANCZOS)
        
    if transform:
        image = transform(image).unsqueeze(0)
        
    return image.to(device)


transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                        std=[0.229, 0.224, 0.225])
]) # 来自ImageNet的mean和variance

content = load_image("png/content.png", transform, max_size=400)
stype = load_image("png/style.png", transform, shape=[content.size(2), content.size(3)])

# content = load_image("png/content.png", transforms.Compose([
#     transforms.ToTensor(),
# ]), max_size=400)
# style = load_image("png/style.png", transforms.Compose([
#     transforms.ToTensor(),
# ]), shape=[content.size(2), content.size(3)])

1	stype.shape

torch.Size([1, 3, 400, 272])

unloader = transforms.ToPILImage()  # reconvert into PIL image

plt.ion()

def imshow(tensor, title=None):
    image = tensor.cpu().clone()  # we clone the tensor to not do changes on it
    image = image.squeeze(0)      # remove the fake batch dimension
    image = unloader(image)
    plt.imshow(image)
    if title is not None:
        plt.title(title)
    plt.pause(0.001) # pause a bit so that plots are updated


plt.figure()
imshow(style[0], title='Image')
# content.shape

png

class VGGNet(nn.Module):
    def __init__(self):
        super(VGGNet, self).__init__()
        self.select = ['0', '5', '10', '19', '28']
        self.vgg = models.vgg19(pretrained=True).features
        
    def forward(self, x):
        features = []
        for name, layer in self.vgg._modules.items():
            x = layer(x)
            if name in self.select:
                features.append(x)
        return features


target = content.clone().requires_grad_(True)
optimizer = torch.optim.Adam([target], lr=0.003, betas=[0.5, 0.999])
vgg = VGGNet().to(device).eval()

1	target_features = vgg(target)

total_step = 2000
style_weight = 100.
for step in range(total_step):
    target_features = vgg(target)
    content_features = vgg(content)
    style_features = vgg(style)
    
    style_loss = 0
    content_loss = 0
    for f1, f2, f3 in zip(target_features, content_features, style_features):
        content_loss += torch.mean((f1-f2)**2)
        _, c, h, w = f1.size()
        f1 = f1.view(c, h*w)
        f3 = f3.view(c, h*w)
        
        # 计算gram matrix
        f1 = torch.mm(f1, f1.t())
        f3 = torch.mm(f3, f3.t())
        style_loss += torch.mean((f1-f3)**2)/(c*h*w)
        
    loss = content_loss + style_weight * style_loss
    
    # 更新target
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if step % 10 == 0:
        print("Step [{}/{}], Content Loss: {:.4f}, Style Loss: {:.4f}"
             .format(step, total_step, content_loss.item(), style_loss.item()))

Step [0/2000], Content Loss: 0.0000, Style Loss: 531.1730
Step [10/2000], Content Loss: 6.0654, Style Loss: 360.6187
Step [20/2000], Content Loss: 11.3430, Style Loss: 253.8006
Step [30/2000], Content Loss: 14.5195, Style Loss: 190.0798
Step [40/2000], Content Loss: 16.5578, Style Loss: 152.3939
Step [50/2000], Content Loss: 17.9683, Style Loss: 129.4922
Step [60/2000], Content Loss: 19.0225, Style Loss: 114.5218
Step [70/2000], Content Loss: 19.8584, Style Loss: 103.7824
Step [80/2000], Content Loss: 20.5509, Style Loss: 95.5047
Step [90/2000], Content Loss: 21.1601, Style Loss: 88.7919
Step [100/2000], Content Loss: 21.6844, Style Loss: 83.1393
Step [110/2000], Content Loss: 22.1447, Style Loss: 78.2809
Step [120/2000], Content Loss: 22.5605, Style Loss: 74.0401
Step [130/2000], Content Loss: 22.9415, Style Loss: 70.2842
Step [140/2000], Content Loss: 23.2941, Style Loss: 66.9353
Step [150/2000], Content Loss: 23.6130, Style Loss: 63.9158
Step [160/2000], Content Loss: 23.9114, Style Loss: 61.1637
Step [170/2000], Content Loss: 24.1892, Style Loss: 58.6509
Step [180/2000], Content Loss: 24.4448, Style Loss: 56.3407
Step [190/2000], Content Loss: 24.6883, Style Loss: 54.1998
Step [200/2000], Content Loss: 24.9212, Style Loss: 52.2185
Step [210/2000], Content Loss: 25.1355, Style Loss: 50.3827
Step [220/2000], Content Loss: 25.3350, Style Loss: 48.6758
Step [230/2000], Content Loss: 25.5269, Style Loss: 47.0833
Step [240/2000], Content Loss: 25.7123, Style Loss: 45.5909
Step [250/2000], Content Loss: 25.8884, Style Loss: 44.1901
Step [260/2000], Content Loss: 26.0555, Style Loss: 42.8741
Step [270/2000], Content Loss: 26.2152, Style Loss: 41.6320
Step [280/2000], Content Loss: 26.3691, Style Loss: 40.4600
Step [290/2000], Content Loss: 26.5208, Style Loss: 39.3519
Step [300/2000], Content Loss: 26.6641, Style Loss: 38.3040
Step [310/2000], Content Loss: 26.8034, Style Loss: 37.3103
Step [320/2000], Content Loss: 26.9339, Style Loss: 36.3693
Step [330/2000], Content Loss: 27.0649, Style Loss: 35.4760
Step [340/2000], Content Loss: 27.1923, Style Loss: 34.6284
Step [350/2000], Content Loss: 27.3130, Style Loss: 33.8245
Step [360/2000], Content Loss: 27.4284, Style Loss: 33.0575
Step [370/2000], Content Loss: 27.5356, Style Loss: 32.3269
Step [380/2000], Content Loss: 27.6426, Style Loss: 31.6281
Step [390/2000], Content Loss: 27.7454, Style Loss: 30.9596
Step [400/2000], Content Loss: 27.8430, Style Loss: 30.3200
Step [410/2000], Content Loss: 27.9398, Style Loss: 29.7072
Step [420/2000], Content Loss: 28.0368, Style Loss: 29.1180
Step [430/2000], Content Loss: 28.1289, Style Loss: 28.5518
Step [440/2000], Content Loss: 28.2207, Style Loss: 28.0077
Step [450/2000], Content Loss: 28.3101, Style Loss: 27.4842
Step [460/2000], Content Loss: 28.4016, Style Loss: 26.9804
Step [470/2000], Content Loss: 28.4844, Style Loss: 26.4949
Step [480/2000], Content Loss: 28.5667, Style Loss: 26.0286
Step [490/2000], Content Loss: 28.6440, Style Loss: 25.5799
Step [500/2000], Content Loss: 28.7183, Style Loss: 25.1476
Step [510/2000], Content Loss: 28.7939, Style Loss: 24.7302
Step [520/2000], Content Loss: 28.8708, Style Loss: 24.3261
Step [530/2000], Content Loss: 28.9440, Style Loss: 23.9349
Step [540/2000], Content Loss: 29.0163, Style Loss: 23.5566
Step [550/2000], Content Loss: 29.0864, Style Loss: 23.1890
Step [560/2000], Content Loss: 29.1529, Style Loss: 22.8329
Step [570/2000], Content Loss: 29.2189, Style Loss: 22.4880
Step [580/2000], Content Loss: 29.2833, Style Loss: 22.1529
Step [590/2000], Content Loss: 29.3477, Style Loss: 21.8286
Step [600/2000], Content Loss: 29.4093, Style Loss: 21.5141
Step [610/2000], Content Loss: 29.4694, Style Loss: 21.2083
Step [620/2000], Content Loss: 29.5252, Style Loss: 20.9107
Step [630/2000], Content Loss: 29.5821, Style Loss: 20.6206
Step [640/2000], Content Loss: 29.6378, Style Loss: 20.3381
Step [650/2000], Content Loss: 29.6938, Style Loss: 20.0623
Step [660/2000], Content Loss: 29.7449, Style Loss: 19.7930
Step [670/2000], Content Loss: 29.7975, Style Loss: 19.5310
Step [680/2000], Content Loss: 29.8479, Style Loss: 19.2760
Step [690/2000], Content Loss: 29.8950, Style Loss: 19.0278
Step [700/2000], Content Loss: 29.9427, Style Loss: 18.7856
Step [710/2000], Content Loss: 29.9889, Style Loss: 18.5502
Step [720/2000], Content Loss: 30.0369, Style Loss: 18.3209
Step [730/2000], Content Loss: 30.0841, Style Loss: 18.0967
Step [740/2000], Content Loss: 30.1312, Style Loss: 17.8776
Step [750/2000], Content Loss: 30.1793, Style Loss: 17.6630
Step [760/2000], Content Loss: 30.2209, Style Loss: 17.4535
Step [770/2000], Content Loss: 30.2625, Style Loss: 17.2486
Step [780/2000], Content Loss: 30.3043, Style Loss: 17.0483
Step [790/2000], Content Loss: 30.3472, Style Loss: 16.8526
Step [800/2000], Content Loss: 30.3883, Style Loss: 16.6612
Step [810/2000], Content Loss: 30.4279, Style Loss: 16.4737
Step [820/2000], Content Loss: 30.4663, Style Loss: 16.2899
Step [830/2000], Content Loss: 30.5036, Style Loss: 16.1099
Step [840/2000], Content Loss: 30.5427, Style Loss: 15.9336
Step [850/2000], Content Loss: 30.5801, Style Loss: 15.7608
Step [860/2000], Content Loss: 30.6190, Style Loss: 15.5913
Step [870/2000], Content Loss: 30.6561, Style Loss: 15.4249
Step [880/2000], Content Loss: 30.6927, Style Loss: 15.2619
Step [890/2000], Content Loss: 30.7275, Style Loss: 15.1023
Step [900/2000], Content Loss: 30.7620, Style Loss: 14.9457
Step [910/2000], Content Loss: 30.7954, Style Loss: 14.7917
Step [920/2000], Content Loss: 30.8298, Style Loss: 14.6399
Step [930/2000], Content Loss: 30.8670, Style Loss: 14.4906
Step [940/2000], Content Loss: 30.9016, Style Loss: 14.3440
Step [950/2000], Content Loss: 30.9369, Style Loss: 14.1998
Step [960/2000], Content Loss: 30.9720, Style Loss: 14.0581
Step [970/2000], Content Loss: 31.0021, Style Loss: 13.9193
Step [980/2000], Content Loss: 31.0370, Style Loss: 13.7825
Step [990/2000], Content Loss: 31.0691, Style Loss: 13.6480
Step [1000/2000], Content Loss: 31.0998, Style Loss: 13.5158
Step [1010/2000], Content Loss: 31.1302, Style Loss: 13.3861
Step [1020/2000], Content Loss: 31.1605, Style Loss: 13.2587
Step [1030/2000], Content Loss: 31.1915, Style Loss: 13.1332
Step [1040/2000], Content Loss: 31.2220, Style Loss: 13.0099
Step [1050/2000], Content Loss: 31.2528, Style Loss: 12.8889
Step [1060/2000], Content Loss: 31.2860, Style Loss: 12.7697
Step [1070/2000], Content Loss: 31.3174, Style Loss: 12.6525
Step [1080/2000], Content Loss: 31.3475, Style Loss: 12.5375
Step [1090/2000], Content Loss: 31.3775, Style Loss: 12.4245
Step [1100/2000], Content Loss: 31.4046, Style Loss: 12.3129
Step [1110/2000], Content Loss: 31.4350, Style Loss: 12.2038
Step [1120/2000], Content Loss: 31.4598, Style Loss: 12.0956
Step [1130/2000], Content Loss: 31.4878, Style Loss: 11.9894
Step [1140/2000], Content Loss: 31.5149, Style Loss: 11.8847
Step [1150/2000], Content Loss: 31.5406, Style Loss: 11.7818
Step [1160/2000], Content Loss: 31.5659, Style Loss: 11.6805
Step [1170/2000], Content Loss: 31.5901, Style Loss: 11.5803
Step [1180/2000], Content Loss: 31.6137, Style Loss: 11.4822
Step [1190/2000], Content Loss: 31.6345, Style Loss: 11.3851
Step [1200/2000], Content Loss: 31.6543, Style Loss: 11.2900
Step [1210/2000], Content Loss: 31.6787, Style Loss: 11.1968
Step [1220/2000], Content Loss: 31.7000, Style Loss: 11.1037
Step [1230/2000], Content Loss: 31.7205, Style Loss: 11.0116
Step [1240/2000], Content Loss: 31.7422, Style Loss: 10.9210
Step [1250/2000], Content Loss: 31.7633, Style Loss: 10.8319
Step [1260/2000], Content Loss: 31.7867, Style Loss: 10.7446
Step [1270/2000], Content Loss: 31.8046, Style Loss: 10.6565
Step [1280/2000], Content Loss: 31.8247, Style Loss: 10.5699
Step [1290/2000], Content Loss: 31.8469, Style Loss: 10.4858
Step [1300/2000], Content Loss: 31.8646, Style Loss: 10.4015
Step [1310/2000], Content Loss: 31.8859, Style Loss: 10.3201
Step [1320/2000], Content Loss: 31.9010, Style Loss: 10.2365
Step [1330/2000], Content Loss: 31.9236, Style Loss: 10.1575
Step [1340/2000], Content Loss: 31.9461, Style Loss: 10.0792
Step [1350/2000], Content Loss: 31.9616, Style Loss: 9.9980
Step [1360/2000], Content Loss: 31.9880, Style Loss: 9.9236
Step [1370/2000], Content Loss: 32.0038, Style Loss: 9.8461
Step [1380/2000], Content Loss: 32.0191, Style Loss: 9.7687
Step [1390/2000], Content Loss: 32.0434, Style Loss: 9.6970
Step [1400/2000], Content Loss: 32.0572, Style Loss: 9.6203
Step [1410/2000], Content Loss: 32.0787, Style Loss: 9.5496
Step [1420/2000], Content Loss: 32.0955, Style Loss: 9.4771
Step [1430/2000], Content Loss: 32.1123, Style Loss: 9.4056
Step [1440/2000], Content Loss: 32.1289, Style Loss: 9.3349
Step [1450/2000], Content Loss: 32.1441, Style Loss: 9.2636
Step [1460/2000], Content Loss: 32.1628, Style Loss: 9.1949
Step [1470/2000], Content Loss: 32.1851, Style Loss: 9.1302
Step [1480/2000], Content Loss: 32.1958, Style Loss: 9.0589
Step [1490/2000], Content Loss: 32.2141, Style Loss: 8.9938
Step [1500/2000], Content Loss: 32.2303, Style Loss: 8.9282
Step [1510/2000], Content Loss: 32.2414, Style Loss: 8.8597
Step [1520/2000], Content Loss: 32.2560, Style Loss: 8.7944
Step [1530/2000], Content Loss: 32.2785, Style Loss: 8.7337
Step [1540/2000], Content Loss: 32.2986, Style Loss: 8.6751
Step [1550/2000], Content Loss: 32.2955, Style Loss: 8.6001
Step [1560/2000], Content Loss: 32.3232, Style Loss: 8.5438
Step [1570/2000], Content Loss: 32.3409, Style Loss: 8.4860
Step [1580/2000], Content Loss: 32.3442, Style Loss: 8.4177
Step [1590/2000], Content Loss: 32.3604, Style Loss: 8.3581
Step [1600/2000], Content Loss: 32.3871, Style Loss: 8.3062
Step [1610/2000], Content Loss: 32.3841, Style Loss: 8.2353
Step [1620/2000], Content Loss: 32.4114, Style Loss: 8.1829
Step [1630/2000], Content Loss: 32.4267, Style Loss: 8.1247
Step [1640/2000], Content Loss: 32.4401, Style Loss: 8.0669
Step [1650/2000], Content Loss: 32.4480, Style Loss: 8.0066
Step [1660/2000], Content Loss: 32.4796, Style Loss: 7.9656
Step [1670/2000], Content Loss: 32.4754, Style Loss: 7.8967
Step [1680/2000], Content Loss: 32.4839, Style Loss: 7.8374
Step [1690/2000], Content Loss: 32.5063, Style Loss: 7.7878
Step [1700/2000], Content Loss: 32.5246, Style Loss: 7.7381
Step [1710/2000], Content Loss: 32.5257, Style Loss: 7.6759
Step [1720/2000], Content Loss: 32.5456, Style Loss: 7.6262
Step [1730/2000], Content Loss: 32.5680, Style Loss: 7.5811
Step [1740/2000], Content Loss: 32.5655, Style Loss: 7.5176
Step [1750/2000], Content Loss: 32.5831, Style Loss: 7.4672
Step [1760/2000], Content Loss: 32.6070, Style Loss: 7.4232
Step [1770/2000], Content Loss: 32.6441, Style Loss: 7.4071
Step [1780/2000], Content Loss: 32.6931, Style Loss: 7.4527
Step [1790/2000], Content Loss: 32.7056, Style Loss: 7.4441
Step [1800/2000], Content Loss: 32.6304, Style Loss: 7.2250
Step [1810/2000], Content Loss: 32.6647, Style Loss: 7.1710
Step [1820/2000], Content Loss: 32.6658, Style Loss: 7.1150
Step [1830/2000], Content Loss: 32.6795, Style Loss: 7.0659
Step [1840/2000], Content Loss: 32.6897, Style Loss: 7.0176
Step [1850/2000], Content Loss: 32.7024, Style Loss: 6.9711
Step [1860/2000], Content Loss: 32.7121, Style Loss: 6.9235
Step [1870/2000], Content Loss: 32.7327, Style Loss: 6.8816
Step [1880/2000], Content Loss: 32.7356, Style Loss: 6.8324
Step [1890/2000], Content Loss: 32.7485, Style Loss: 6.7878
Step [1900/2000], Content Loss: 32.7634, Style Loss: 6.7444
Step [1910/2000], Content Loss: 32.7753, Style Loss: 6.6990
Step [1920/2000], Content Loss: 32.7872, Style Loss: 6.6547
Step [1930/2000], Content Loss: 32.8038, Style Loss: 6.6145
Step [1940/2000], Content Loss: 32.8169, Style Loss: 6.5722
Step [1950/2000], Content Loss: 32.8173, Style Loss: 6.5240
Step [1960/2000], Content Loss: 32.8359, Style Loss: 6.4847
Step [1970/2000], Content Loss: 32.8538, Style Loss: 6.4470
Step [1980/2000], Content Loss: 32.8599, Style Loss: 6.4017
Step [1990/2000], Content Loss: 32.8634, Style Loss: 6.3566

denorm = transforms.Normalize((-2.12, -2.04, -1.80), (4.37, 4.46, 4.44))
img = target.clone().squeeze()
img = denorm(img).clamp_(0, 1)
plt.figure()
imshow(img, title='Target Image')

png

Generative Adversarial Networks

batch_size=32
transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=(0.5, 0.5, 0.5),
                        std=(0.5, 0.5, 0.5))
])

mnist_data = torchvision.datasets.MNIST("./mnist_data", train=True, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(dataset=mnist_data,
                                         batch_size=batch_size,
                                         shuffle=True)

image_size = 784

hidden_size = 256
# discriminator
D = nn.Sequential(
    nn.Linear(image_size, hidden_size),
    nn.LeakyReLU(0.2),
    nn.Linear(hidden_size, hidden_size),
    nn.LeakyReLU(0.2),
    nn.Linear(hidden_size, 1),
    nn.Sigmoid()
)

latent_size = 64
# Generator
G = nn.Sequential(
    nn.Linear(latent_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, hidden_size),
    nn.ReLU(),
    nn.Linear(hidden_size, image_size),
    nn.Tanh()
)

D = D.to(device)
G = G.to(device)

loss_fn = nn.BCELoss()
d_optimizer = torch.optim.Adam(D.parameters(), lr=0.0002)
g_optimizer = torch.optim.Adam(G.parameters(), lr=0.0002)

开始训练


def reset_grad():
    d_optimizer.zero_grad()
    g_optimizer.zero_grad()

total_step = len(dataloader)
num_epochs = 200
for epoch in range(num_epochs):
    for i, (images, _) in enumerate(dataloader):
        batch_size = images.size(0)
        images = images.reshape(batch_size, image_size).to(device)
        
        real_labels = torch.ones(batch_size, 1).to(device)
        fake_labels = torch.zeros(batch_size, 1).to(device)
        
        outputs = D(images)
        d_loss_real = loss_fn(outputs, real_labels)
        real_score = outputs
        
        # 开始生成fake images
        z = torch.randn(batch_size, latent_size).to(device)
        fake_images = G(z)
        outputs = D(fake_images.detach())
        d_loss_fake = loss_fn(outputs, fake_labels)
        fake_score = outputs
        
        # 开始优化discriminator
        d_loss = d_loss_real + d_loss_fake
        reset_grad()
        d_loss.backward()
        d_optimizer.step()
        
        # 开始优化generator
        z = torch.randn(batch_size, latent_size).to(device)
        fake_images = G(z)
        outputs = D(fake_images)
        g_loss = loss_fn(outputs, real_labels)
        
        reset_grad()
        g_loss.backward()
        g_optimizer.step()
        
        if i % 1000 == 0:
            print("Epoch [{}/{}], Step [{}/{}], d_loss: {:.4f}, g_loss: {:.4f}, D(x): {:.2f}, D(G(z)): {:.2f}"
                 .format(epoch, num_epochs, i, total_step, d_loss.item(), g_loss.item(), real_score.mean().item(), fake_score.mean().item()))

Epoch [0/200], Step [0/1875], d_loss: 0.6669, g_loss: 2.9577, D(x): 0.76, D(G(z)): 0.15
Epoch [0/200], Step [1000/1875], d_loss: 0.1716, g_loss: 3.0008, D(x): 0.93, D(G(z)): 0.09
Epoch [1/200], Step [0/1875], d_loss: 0.1716, g_loss: 4.1396, D(x): 0.93, D(G(z)): 0.02
Epoch [1/200], Step [1000/1875], d_loss: 0.0202, g_loss: 5.1296, D(x): 1.00, D(G(z)): 0.02
Epoch [2/200], Step [0/1875], d_loss: 0.2070, g_loss: 3.7713, D(x): 0.93, D(G(z)): 0.08
Epoch [2/200], Step [1000/1875], d_loss: 0.0829, g_loss: 4.9163, D(x): 0.99, D(G(z)): 0.07
Epoch [3/200], Step [0/1875], d_loss: 0.2986, g_loss: 3.6197, D(x): 0.90, D(G(z)): 0.03
Epoch [3/200], Step [1000/1875], d_loss: 0.4204, g_loss: 2.2956, D(x): 0.90, D(G(z)): 0.14
Epoch [4/200], Step [0/1875], d_loss: 0.4453, g_loss: 5.1677, D(x): 0.80, D(G(z)): 0.02
Epoch [4/200], Step [1000/1875], d_loss: 0.1900, g_loss: 2.7722, D(x): 0.93, D(G(z)): 0.10
Epoch [5/200], Step [0/1875], d_loss: 0.3418, g_loss: 2.4469, D(x): 1.00, D(G(z)): 0.21
Epoch [5/200], Step [1000/1875], d_loss: 0.4460, g_loss: 2.4152, D(x): 0.90, D(G(z)): 0.18
Epoch [6/200], Step [0/1875], d_loss: 0.3142, g_loss: 4.0145, D(x): 0.93, D(G(z)): 0.13
Epoch [6/200], Step [1000/1875], d_loss: 0.5893, g_loss: 3.9873, D(x): 0.97, D(G(z)): 0.31
Epoch [7/200], Step [0/1875], d_loss: 0.3118, g_loss: 3.2590, D(x): 0.88, D(G(z)): 0.10
Epoch [7/200], Step [1000/1875], d_loss: 0.5169, g_loss: 2.8562, D(x): 0.84, D(G(z)): 0.20
Epoch [8/200], Step [0/1875], d_loss: 0.1886, g_loss: 3.0765, D(x): 0.93, D(G(z)): 0.05
Epoch [8/200], Step [1000/1875], d_loss: 0.5987, g_loss: 3.0972, D(x): 0.86, D(G(z)): 0.17
Epoch [9/200], Step [0/1875], d_loss: 0.7312, g_loss: 2.5704, D(x): 0.93, D(G(z)): 0.30
Epoch [9/200], Step [1000/1875], d_loss: 0.2202, g_loss: 3.1345, D(x): 0.94, D(G(z)): 0.11
Epoch [10/200], Step [0/1875], d_loss: 0.5448, g_loss: 3.2835, D(x): 0.81, D(G(z)): 0.11
Epoch [10/200], Step [1000/1875], d_loss: 0.4599, g_loss: 2.8296, D(x): 0.81, D(G(z)): 0.09
Epoch [11/200], Step [0/1875], d_loss: 0.3990, g_loss: 3.9110, D(x): 0.86, D(G(z)): 0.11
Epoch [11/200], Step [1000/1875], d_loss: 0.4137, g_loss: 3.2849, D(x): 0.88, D(G(z)): 0.17
Epoch [12/200], Step [0/1875], d_loss: 0.6989, g_loss: 2.1561, D(x): 0.80, D(G(z)): 0.24
Epoch [12/200], Step [1000/1875], d_loss: 0.7982, g_loss: 2.6202, D(x): 0.75, D(G(z)): 0.27
Epoch [13/200], Step [0/1875], d_loss: 0.7775, g_loss: 2.6229, D(x): 0.70, D(G(z)): 0.09
Epoch [13/200], Step [1000/1875], d_loss: 0.7904, g_loss: 2.3377, D(x): 0.69, D(G(z)): 0.06
Epoch [14/200], Step [0/1875], d_loss: 0.5520, g_loss: 3.6026, D(x): 0.87, D(G(z)): 0.23
Epoch [14/200], Step [1000/1875], d_loss: 0.4877, g_loss: 1.8566, D(x): 0.81, D(G(z)): 0.11
Epoch [15/200], Step [0/1875], d_loss: 0.6178, g_loss: 2.8264, D(x): 0.73, D(G(z)): 0.08
Epoch [15/200], Step [1000/1875], d_loss: 0.5656, g_loss: 2.0427, D(x): 0.85, D(G(z)): 0.23
Epoch [16/200], Step [0/1875], d_loss: 0.7704, g_loss: 1.8280, D(x): 0.82, D(G(z)): 0.28
Epoch [16/200], Step [1000/1875], d_loss: 0.4717, g_loss: 2.3330, D(x): 0.87, D(G(z)): 0.23
Epoch [17/200], Step [0/1875], d_loss: 0.6158, g_loss: 2.3867, D(x): 0.80, D(G(z)): 0.21
Epoch [17/200], Step [1000/1875], d_loss: 0.5036, g_loss: 2.1572, D(x): 0.86, D(G(z)): 0.22
Epoch [18/200], Step [0/1875], d_loss: 0.2080, g_loss: 3.1542, D(x): 0.97, D(G(z)): 0.13
Epoch [18/200], Step [1000/1875], d_loss: 0.4262, g_loss: 3.2852, D(x): 0.85, D(G(z)): 0.12
Epoch [19/200], Step [0/1875], d_loss: 1.1834, g_loss: 1.7200, D(x): 0.82, D(G(z)): 0.44
Epoch [19/200], Step [1000/1875], d_loss: 0.7412, g_loss: 3.3823, D(x): 0.70, D(G(z)): 0.14
Epoch [20/200], Step [0/1875], d_loss: 0.8160, g_loss: 2.5552, D(x): 0.78, D(G(z)): 0.28
Epoch [20/200], Step [1000/1875], d_loss: 0.8000, g_loss: 1.9645, D(x): 0.82, D(G(z)): 0.31
Epoch [21/200], Step [0/1875], d_loss: 0.8578, g_loss: 2.7063, D(x): 0.70, D(G(z)): 0.24
Epoch [21/200], Step [1000/1875], d_loss: 0.4567, g_loss: 1.8023, D(x): 0.83, D(G(z)): 0.18
Epoch [22/200], Step [0/1875], d_loss: 0.6396, g_loss: 1.9526, D(x): 0.76, D(G(z)): 0.20
Epoch [22/200], Step [1000/1875], d_loss: 0.4177, g_loss: 2.4358, D(x): 0.89, D(G(z)): 0.18
Epoch [23/200], Step [0/1875], d_loss: 0.7560, g_loss: 2.3783, D(x): 0.83, D(G(z)): 0.36
Epoch [23/200], Step [1000/1875], d_loss: 0.8418, g_loss: 1.2812, D(x): 0.72, D(G(z)): 0.22
Epoch [24/200], Step [0/1875], d_loss: 0.8319, g_loss: 2.1962, D(x): 0.66, D(G(z)): 0.20
Epoch [24/200], Step [1000/1875], d_loss: 0.8614, g_loss: 2.3836, D(x): 0.67, D(G(z)): 0.20
Epoch [25/200], Step [0/1875], d_loss: 0.8590, g_loss: 1.5315, D(x): 0.78, D(G(z)): 0.36
Epoch [25/200], Step [1000/1875], d_loss: 0.9564, g_loss: 1.7998, D(x): 0.77, D(G(z)): 0.38
Epoch [26/200], Step [0/1875], d_loss: 0.8937, g_loss: 1.3713, D(x): 0.74, D(G(z)): 0.31
Epoch [26/200], Step [1000/1875], d_loss: 0.9061, g_loss: 2.4561, D(x): 0.74, D(G(z)): 0.27
Epoch [27/200], Step [0/1875], d_loss: 0.6779, g_loss: 2.1518, D(x): 0.76, D(G(z)): 0.23
Epoch [27/200], Step [1000/1875], d_loss: 1.0955, g_loss: 1.9235, D(x): 0.70, D(G(z)): 0.31
Epoch [28/200], Step [0/1875], d_loss: 0.7943, g_loss: 1.5614, D(x): 0.73, D(G(z)): 0.24
Epoch [28/200], Step [1000/1875], d_loss: 0.8096, g_loss: 1.8443, D(x): 0.86, D(G(z)): 0.40
Epoch [29/200], Step [0/1875], d_loss: 0.6123, g_loss: 1.8900, D(x): 0.80, D(G(z)): 0.23
Epoch [29/200], Step [1000/1875], d_loss: 0.9214, g_loss: 1.5088, D(x): 0.79, D(G(z)): 0.38
Epoch [30/200], Step [0/1875], d_loss: 1.1502, g_loss: 1.2392, D(x): 0.63, D(G(z)): 0.31
Epoch [30/200], Step [1000/1875], d_loss: 0.7820, g_loss: 1.2615, D(x): 0.81, D(G(z)): 0.35
Epoch [31/200], Step [0/1875], d_loss: 0.9985, g_loss: 1.9074, D(x): 0.63, D(G(z)): 0.23
Epoch [31/200], Step [1000/1875], d_loss: 0.7422, g_loss: 1.5258, D(x): 0.72, D(G(z)): 0.26
Epoch [32/200], Step [0/1875], d_loss: 0.9283, g_loss: 2.1753, D(x): 0.60, D(G(z)): 0.20
Epoch [32/200], Step [1000/1875], d_loss: 0.6156, g_loss: 1.8300, D(x): 0.88, D(G(z)): 0.34
Epoch [33/200], Step [0/1875], d_loss: 0.7572, g_loss: 2.5281, D(x): 0.69, D(G(z)): 0.20
Epoch [33/200], Step [1000/1875], d_loss: 1.2556, g_loss: 1.6872, D(x): 0.58, D(G(z)): 0.31
Epoch [34/200], Step [0/1875], d_loss: 0.9278, g_loss: 1.6144, D(x): 0.77, D(G(z)): 0.37
Epoch [34/200], Step [1000/1875], d_loss: 1.0190, g_loss: 1.9249, D(x): 0.65, D(G(z)): 0.31
Epoch [35/200], Step [0/1875], d_loss: 1.1411, g_loss: 1.3005, D(x): 0.79, D(G(z)): 0.47
Epoch [35/200], Step [1000/1875], d_loss: 0.9863, g_loss: 0.9696, D(x): 0.81, D(G(z)): 0.45
Epoch [36/200], Step [0/1875], d_loss: 0.6408, g_loss: 1.7086, D(x): 0.77, D(G(z)): 0.24
Epoch [36/200], Step [1000/1875], d_loss: 0.8755, g_loss: 1.4808, D(x): 0.71, D(G(z)): 0.31
Epoch [37/200], Step [0/1875], d_loss: 0.8984, g_loss: 1.3038, D(x): 0.77, D(G(z)): 0.37
Epoch [37/200], Step [1000/1875], d_loss: 0.8318, g_loss: 1.4391, D(x): 0.73, D(G(z)): 0.29
Epoch [38/200], Step [0/1875], d_loss: 0.6922, g_loss: 1.8307, D(x): 0.77, D(G(z)): 0.27
Epoch [38/200], Step [1000/1875], d_loss: 1.1070, g_loss: 1.1424, D(x): 0.71, D(G(z)): 0.45
Epoch [39/200], Step [0/1875], d_loss: 0.8160, g_loss: 1.7084, D(x): 0.79, D(G(z)): 0.31
Epoch [39/200], Step [1000/1875], d_loss: 0.7833, g_loss: 1.5914, D(x): 0.69, D(G(z)): 0.16
Epoch [40/200], Step [0/1875], d_loss: 1.1307, g_loss: 1.1723, D(x): 0.72, D(G(z)): 0.42
Epoch [40/200], Step [1000/1875], d_loss: 0.9260, g_loss: 1.5115, D(x): 0.58, D(G(z)): 0.19
Epoch [41/200], Step [0/1875], d_loss: 0.8279, g_loss: 2.0445, D(x): 0.71, D(G(z)): 0.26
Epoch [41/200], Step [1000/1875], d_loss: 1.0122, g_loss: 1.4877, D(x): 0.68, D(G(z)): 0.34
Epoch [42/200], Step [0/1875], d_loss: 1.0094, g_loss: 1.5560, D(x): 0.67, D(G(z)): 0.30
Epoch [42/200], Step [1000/1875], d_loss: 1.1574, g_loss: 1.0871, D(x): 0.80, D(G(z)): 0.48
Epoch [43/200], Step [0/1875], d_loss: 0.7671, g_loss: 1.4075, D(x): 0.72, D(G(z)): 0.23
Epoch [43/200], Step [1000/1875], d_loss: 0.8994, g_loss: 1.4649, D(x): 0.69, D(G(z)): 0.28
Epoch [44/200], Step [0/1875], d_loss: 0.8590, g_loss: 1.2829, D(x): 0.75, D(G(z)): 0.35
Epoch [44/200], Step [1000/1875], d_loss: 0.8026, g_loss: 2.1658, D(x): 0.64, D(G(z)): 0.18
Epoch [45/200], Step [0/1875], d_loss: 1.1981, g_loss: 1.5492, D(x): 0.65, D(G(z)): 0.37
Epoch [45/200], Step [1000/1875], d_loss: 1.0184, g_loss: 1.2799, D(x): 0.68, D(G(z)): 0.37
Epoch [46/200], Step [0/1875], d_loss: 0.7981, g_loss: 2.0579, D(x): 0.71, D(G(z)): 0.26
Epoch [46/200], Step [1000/1875], d_loss: 1.1051, g_loss: 1.2950, D(x): 0.63, D(G(z)): 0.28
Epoch [47/200], Step [0/1875], d_loss: 0.9363, g_loss: 1.2712, D(x): 0.64, D(G(z)): 0.26
Epoch [47/200], Step [1000/1875], d_loss: 0.7284, g_loss: 1.2780, D(x): 0.82, D(G(z)): 0.36
Epoch [48/200], Step [0/1875], d_loss: 0.9353, g_loss: 1.6880, D(x): 0.76, D(G(z)): 0.41
Epoch [48/200], Step [1000/1875], d_loss: 0.9996, g_loss: 1.7311, D(x): 0.70, D(G(z)): 0.32
Epoch [49/200], Step [0/1875], d_loss: 0.9926, g_loss: 1.4112, D(x): 0.78, D(G(z)): 0.42
Epoch [49/200], Step [1000/1875], d_loss: 0.8023, g_loss: 1.6557, D(x): 0.65, D(G(z)): 0.21
Epoch [50/200], Step [0/1875], d_loss: 0.8718, g_loss: 1.9058, D(x): 0.63, D(G(z)): 0.20
Epoch [50/200], Step [1000/1875], d_loss: 0.9961, g_loss: 1.5768, D(x): 0.62, D(G(z)): 0.28
Epoch [51/200], Step [0/1875], d_loss: 0.9317, g_loss: 1.3332, D(x): 0.70, D(G(z)): 0.33
Epoch [51/200], Step [1000/1875], d_loss: 0.9427, g_loss: 1.1736, D(x): 0.68, D(G(z)): 0.32
Epoch [52/200], Step [0/1875], d_loss: 0.7741, g_loss: 1.6549, D(x): 0.74, D(G(z)): 0.29
Epoch [52/200], Step [1000/1875], d_loss: 1.2812, g_loss: 1.1068, D(x): 0.71, D(G(z)): 0.49
Epoch [53/200], Step [0/1875], d_loss: 0.8245, g_loss: 1.5040, D(x): 0.73, D(G(z)): 0.28
Epoch [53/200], Step [1000/1875], d_loss: 1.0251, g_loss: 1.2684, D(x): 0.80, D(G(z)): 0.44
Epoch [54/200], Step [0/1875], d_loss: 1.1557, g_loss: 1.8746, D(x): 0.67, D(G(z)): 0.35
Epoch [54/200], Step [1000/1875], d_loss: 1.1738, g_loss: 1.6428, D(x): 0.57, D(G(z)): 0.33
Epoch [55/200], Step [0/1875], d_loss: 1.0400, g_loss: 1.3476, D(x): 0.55, D(G(z)): 0.21
Epoch [55/200], Step [1000/1875], d_loss: 1.0220, g_loss: 1.4821, D(x): 0.59, D(G(z)): 0.24
Epoch [56/200], Step [0/1875], d_loss: 0.7882, g_loss: 1.4944, D(x): 0.66, D(G(z)): 0.20
Epoch [56/200], Step [1000/1875], d_loss: 0.8876, g_loss: 1.5311, D(x): 0.73, D(G(z)): 0.34
Epoch [57/200], Step [0/1875], d_loss: 1.0530, g_loss: 1.7741, D(x): 0.69, D(G(z)): 0.36
Epoch [57/200], Step [1000/1875], d_loss: 1.1232, g_loss: 1.5487, D(x): 0.62, D(G(z)): 0.33
Epoch [58/200], Step [0/1875], d_loss: 1.0350, g_loss: 1.3535, D(x): 0.73, D(G(z)): 0.43
Epoch [58/200], Step [1000/1875], d_loss: 0.7528, g_loss: 1.4546, D(x): 0.74, D(G(z)): 0.28
Epoch [59/200], Step [0/1875], d_loss: 0.9243, g_loss: 1.3529, D(x): 0.71, D(G(z)): 0.31
Epoch [59/200], Step [1000/1875], d_loss: 1.0429, g_loss: 1.6492, D(x): 0.64, D(G(z)): 0.36
Epoch [60/200], Step [0/1875], d_loss: 0.9420, g_loss: 1.4876, D(x): 0.68, D(G(z)): 0.31
Epoch [60/200], Step [1000/1875], d_loss: 1.0196, g_loss: 1.6513, D(x): 0.67, D(G(z)): 0.34
Epoch [61/200], Step [0/1875], d_loss: 1.0662, g_loss: 1.4362, D(x): 0.64, D(G(z)): 0.29
Epoch [61/200], Step [1000/1875], d_loss: 1.1993, g_loss: 1.2304, D(x): 0.55, D(G(z)): 0.34
Epoch [62/200], Step [0/1875], d_loss: 1.1418, g_loss: 1.6582, D(x): 0.57, D(G(z)): 0.27
Epoch [62/200], Step [1000/1875], d_loss: 1.1739, g_loss: 1.0282, D(x): 0.72, D(G(z)): 0.50
Epoch [63/200], Step [0/1875], d_loss: 0.9645, g_loss: 1.2030, D(x): 0.67, D(G(z)): 0.32
Epoch [63/200], Step [1000/1875], d_loss: 1.0324, g_loss: 1.8831, D(x): 0.63, D(G(z)): 0.30
Epoch [64/200], Step [0/1875], d_loss: 1.2073, g_loss: 1.2013, D(x): 0.60, D(G(z)): 0.37
Epoch [64/200], Step [1000/1875], d_loss: 1.3382, g_loss: 1.4971, D(x): 0.69, D(G(z)): 0.47
Epoch [65/200], Step [0/1875], d_loss: 0.7616, g_loss: 1.4244, D(x): 0.79, D(G(z)): 0.33
Epoch [65/200], Step [1000/1875], d_loss: 0.9834, g_loss: 1.9160, D(x): 0.60, D(G(z)): 0.24
Epoch [66/200], Step [0/1875], d_loss: 0.9860, g_loss: 1.2135, D(x): 0.71, D(G(z)): 0.36
Epoch [66/200], Step [1000/1875], d_loss: 1.1599, g_loss: 1.9320, D(x): 0.56, D(G(z)): 0.24
Epoch [67/200], Step [0/1875], d_loss: 0.9280, g_loss: 1.6222, D(x): 0.62, D(G(z)): 0.25
Epoch [67/200], Step [1000/1875], d_loss: 0.8609, g_loss: 1.2151, D(x): 0.72, D(G(z)): 0.34
Epoch [68/200], Step [0/1875], d_loss: 1.1169, g_loss: 1.2863, D(x): 0.64, D(G(z)): 0.35
Epoch [68/200], Step [1000/1875], d_loss: 1.3884, g_loss: 1.1648, D(x): 0.80, D(G(z)): 0.59
Epoch [69/200], Step [0/1875], d_loss: 0.7709, g_loss: 1.5080, D(x): 0.72, D(G(z)): 0.29
Epoch [69/200], Step [1000/1875], d_loss: 0.9492, g_loss: 1.4181, D(x): 0.67, D(G(z)): 0.29
Epoch [70/200], Step [0/1875], d_loss: 0.8738, g_loss: 1.2650, D(x): 0.74, D(G(z)): 0.36
Epoch [70/200], Step [1000/1875], d_loss: 1.0756, g_loss: 1.4710, D(x): 0.74, D(G(z)): 0.41
Epoch [71/200], Step [0/1875], d_loss: 0.8898, g_loss: 1.4363, D(x): 0.69, D(G(z)): 0.30
Epoch [71/200], Step [1000/1875], d_loss: 0.9169, g_loss: 1.2323, D(x): 0.63, D(G(z)): 0.25
Epoch [72/200], Step [0/1875], d_loss: 0.9560, g_loss: 1.2931, D(x): 0.63, D(G(z)): 0.29
Epoch [72/200], Step [1000/1875], d_loss: 0.9121, g_loss: 1.6194, D(x): 0.69, D(G(z)): 0.30
Epoch [73/200], Step [0/1875], d_loss: 0.9210, g_loss: 1.6881, D(x): 0.64, D(G(z)): 0.29
Epoch [73/200], Step [1000/1875], d_loss: 0.9212, g_loss: 1.6392, D(x): 0.72, D(G(z)): 0.36
Epoch [74/200], Step [0/1875], d_loss: 1.2269, g_loss: 1.4554, D(x): 0.57, D(G(z)): 0.35
Epoch [74/200], Step [1000/1875], d_loss: 1.0380, g_loss: 1.3137, D(x): 0.81, D(G(z)): 0.46
Epoch [75/200], Step [0/1875], d_loss: 1.0824, g_loss: 2.1083, D(x): 0.60, D(G(z)): 0.28
Epoch [75/200], Step [1000/1875], d_loss: 1.0364, g_loss: 1.2388, D(x): 0.61, D(G(z)): 0.32
Epoch [76/200], Step [0/1875], d_loss: 1.0572, g_loss: 1.7266, D(x): 0.58, D(G(z)): 0.24
Epoch [76/200], Step [1000/1875], d_loss: 1.1760, g_loss: 1.3603, D(x): 0.66, D(G(z)): 0.39
Epoch [77/200], Step [0/1875], d_loss: 0.7916, g_loss: 1.2981, D(x): 0.71, D(G(z)): 0.27
Epoch [77/200], Step [1000/1875], d_loss: 0.9169, g_loss: 1.3591, D(x): 0.68, D(G(z)): 0.32
Epoch [78/200], Step [0/1875], d_loss: 0.9650, g_loss: 1.2724, D(x): 0.70, D(G(z)): 0.39
Epoch [78/200], Step [1000/1875], d_loss: 1.0706, g_loss: 1.5743, D(x): 0.72, D(G(z)): 0.41
Epoch [79/200], Step [0/1875], d_loss: 1.0080, g_loss: 1.4655, D(x): 0.61, D(G(z)): 0.30
Epoch [79/200], Step [1000/1875], d_loss: 0.9786, g_loss: 1.2689, D(x): 0.67, D(G(z)): 0.36
Epoch [80/200], Step [0/1875], d_loss: 0.9673, g_loss: 1.3955, D(x): 0.80, D(G(z)): 0.44
Epoch [80/200], Step [1000/1875], d_loss: 1.0951, g_loss: 1.3826, D(x): 0.71, D(G(z)): 0.43
Epoch [81/200], Step [0/1875], d_loss: 0.9750, g_loss: 1.7231, D(x): 0.55, D(G(z)): 0.21
Epoch [81/200], Step [1000/1875], d_loss: 0.8631, g_loss: 1.3905, D(x): 0.69, D(G(z)): 0.31
Epoch [82/200], Step [0/1875], d_loss: 1.2233, g_loss: 1.3238, D(x): 0.61, D(G(z)): 0.39
Epoch [82/200], Step [1000/1875], d_loss: 1.1894, g_loss: 1.3720, D(x): 0.70, D(G(z)): 0.40
Epoch [83/200], Step [0/1875], d_loss: 1.0209, g_loss: 1.5222, D(x): 0.63, D(G(z)): 0.26
Epoch [83/200], Step [1000/1875], d_loss: 0.7264, g_loss: 1.6697, D(x): 0.73, D(G(z)): 0.28
Epoch [84/200], Step [0/1875], d_loss: 0.8771, g_loss: 1.2173, D(x): 0.76, D(G(z)): 0.36
Epoch [84/200], Step [1000/1875], d_loss: 0.9297, g_loss: 1.4891, D(x): 0.72, D(G(z)): 0.34
Epoch [85/200], Step [0/1875], d_loss: 0.9688, g_loss: 1.7294, D(x): 0.63, D(G(z)): 0.29
Epoch [85/200], Step [1000/1875], d_loss: 0.8822, g_loss: 1.5354, D(x): 0.70, D(G(z)): 0.30
Epoch [86/200], Step [0/1875], d_loss: 1.1903, g_loss: 1.2292, D(x): 0.71, D(G(z)): 0.45
Epoch [86/200], Step [1000/1875], d_loss: 1.0713, g_loss: 1.3514, D(x): 0.67, D(G(z)): 0.39
Epoch [87/200], Step [0/1875], d_loss: 0.9523, g_loss: 1.4799, D(x): 0.63, D(G(z)): 0.26
Epoch [87/200], Step [1000/1875], d_loss: 1.1543, g_loss: 1.3191, D(x): 0.63, D(G(z)): 0.35
Epoch [88/200], Step [0/1875], d_loss: 1.0270, g_loss: 1.3444, D(x): 0.63, D(G(z)): 0.33
Epoch [88/200], Step [1000/1875], d_loss: 0.9212, g_loss: 1.6030, D(x): 0.60, D(G(z)): 0.23
Epoch [89/200], Step [0/1875], d_loss: 1.1040, g_loss: 1.2642, D(x): 0.64, D(G(z)): 0.34
Epoch [89/200], Step [1000/1875], d_loss: 0.8394, g_loss: 1.4969, D(x): 0.75, D(G(z)): 0.34
Epoch [90/200], Step [0/1875], d_loss: 0.9523, g_loss: 0.9641, D(x): 0.79, D(G(z)): 0.40
Epoch [90/200], Step [1000/1875], d_loss: 0.7576, g_loss: 1.0150, D(x): 0.78, D(G(z)): 0.33
Epoch [91/200], Step [0/1875], d_loss: 1.2105, g_loss: 0.9780, D(x): 0.66, D(G(z)): 0.41
Epoch [91/200], Step [1000/1875], d_loss: 1.0656, g_loss: 1.5340, D(x): 0.60, D(G(z)): 0.32
Epoch [92/200], Step [0/1875], d_loss: 0.9305, g_loss: 1.5715, D(x): 0.64, D(G(z)): 0.28
Epoch [92/200], Step [1000/1875], d_loss: 0.8817, g_loss: 1.5210, D(x): 0.71, D(G(z)): 0.31
Epoch [93/200], Step [0/1875], d_loss: 0.8735, g_loss: 1.8431, D(x): 0.62, D(G(z)): 0.23
Epoch [93/200], Step [1000/1875], d_loss: 1.2207, g_loss: 1.4299, D(x): 0.61, D(G(z)): 0.36
Epoch [94/200], Step [0/1875], d_loss: 1.1631, g_loss: 1.6790, D(x): 0.53, D(G(z)): 0.25
Epoch [94/200], Step [1000/1875], d_loss: 1.0503, g_loss: 1.3590, D(x): 0.67, D(G(z)): 0.37
Epoch [95/200], Step [0/1875], d_loss: 0.9073, g_loss: 1.3901, D(x): 0.65, D(G(z)): 0.29
Epoch [95/200], Step [1000/1875], d_loss: 0.9264, g_loss: 1.4881, D(x): 0.70, D(G(z)): 0.36
Epoch [96/200], Step [0/1875], d_loss: 0.8375, g_loss: 1.6237, D(x): 0.68, D(G(z)): 0.28
Epoch [96/200], Step [1000/1875], d_loss: 0.8759, g_loss: 1.6055, D(x): 0.70, D(G(z)): 0.32
Epoch [97/200], Step [0/1875], d_loss: 0.9862, g_loss: 1.2774, D(x): 0.73, D(G(z)): 0.42
Epoch [97/200], Step [1000/1875], d_loss: 0.8995, g_loss: 1.3931, D(x): 0.64, D(G(z)): 0.29
Epoch [98/200], Step [0/1875], d_loss: 1.1893, g_loss: 1.0463, D(x): 0.76, D(G(z)): 0.46
Epoch [98/200], Step [1000/1875], d_loss: 1.0180, g_loss: 1.0250, D(x): 0.58, D(G(z)): 0.26
Epoch [99/200], Step [0/1875], d_loss: 0.7713, g_loss: 1.3374, D(x): 0.70, D(G(z)): 0.26
Epoch [99/200], Step [1000/1875], d_loss: 0.9064, g_loss: 1.0758, D(x): 0.74, D(G(z)): 0.37
Epoch [100/200], Step [0/1875], d_loss: 1.0002, g_loss: 1.2143, D(x): 0.64, D(G(z)): 0.30
Epoch [100/200], Step [1000/1875], d_loss: 1.0911, g_loss: 1.3313, D(x): 0.70, D(G(z)): 0.41
Epoch [101/200], Step [0/1875], d_loss: 0.8495, g_loss: 1.9575, D(x): 0.62, D(G(z)): 0.19
Epoch [101/200], Step [1000/1875], d_loss: 0.8246, g_loss: 1.1735, D(x): 0.72, D(G(z)): 0.33
Epoch [102/200], Step [0/1875], d_loss: 0.8016, g_loss: 1.5931, D(x): 0.68, D(G(z)): 0.23
Epoch [102/200], Step [1000/1875], d_loss: 0.7966, g_loss: 1.5136, D(x): 0.65, D(G(z)): 0.22
Epoch [103/200], Step [0/1875], d_loss: 0.8603, g_loss: 1.2868, D(x): 0.80, D(G(z)): 0.39
Epoch [103/200], Step [1000/1875], d_loss: 0.9518, g_loss: 1.7202, D(x): 0.74, D(G(z)): 0.37
Epoch [104/200], Step [0/1875], d_loss: 0.7930, g_loss: 1.7609, D(x): 0.73, D(G(z)): 0.30
Epoch [104/200], Step [1000/1875], d_loss: 1.2606, g_loss: 1.1577, D(x): 0.66, D(G(z)): 0.44
Epoch [105/200], Step [0/1875], d_loss: 1.0098, g_loss: 1.5430, D(x): 0.65, D(G(z)): 0.34
Epoch [105/200], Step [1000/1875], d_loss: 0.9373, g_loss: 0.9949, D(x): 0.76, D(G(z)): 0.39
Epoch [106/200], Step [0/1875], d_loss: 0.9693, g_loss: 1.5791, D(x): 0.68, D(G(z)): 0.34
Epoch [106/200], Step [1000/1875], d_loss: 0.9154, g_loss: 1.4726, D(x): 0.73, D(G(z)): 0.31
Epoch [107/200], Step [0/1875], d_loss: 0.9514, g_loss: 1.7878, D(x): 0.60, D(G(z)): 0.22
Epoch [107/200], Step [1000/1875], d_loss: 1.0044, g_loss: 1.4046, D(x): 0.63, D(G(z)): 0.30
Epoch [108/200], Step [0/1875], d_loss: 0.8615, g_loss: 1.6039, D(x): 0.63, D(G(z)): 0.23
Epoch [108/200], Step [1000/1875], d_loss: 0.8843, g_loss: 1.9490, D(x): 0.62, D(G(z)): 0.25
Epoch [109/200], Step [0/1875], d_loss: 1.0323, g_loss: 1.4124, D(x): 0.63, D(G(z)): 0.32
Epoch [109/200], Step [1000/1875], d_loss: 0.8610, g_loss: 1.2935, D(x): 0.75, D(G(z)): 0.34
Epoch [110/200], Step [0/1875], d_loss: 1.1965, g_loss: 1.6509, D(x): 0.54, D(G(z)): 0.31
Epoch [110/200], Step [1000/1875], d_loss: 0.9098, g_loss: 1.0422, D(x): 0.69, D(G(z)): 0.27
Epoch [111/200], Step [0/1875], d_loss: 1.1742, g_loss: 1.7862, D(x): 0.60, D(G(z)): 0.34
Epoch [111/200], Step [1000/1875], d_loss: 1.0664, g_loss: 1.7042, D(x): 0.57, D(G(z)): 0.27
Epoch [112/200], Step [0/1875], d_loss: 0.9700, g_loss: 1.7371, D(x): 0.68, D(G(z)): 0.30
Epoch [112/200], Step [1000/1875], d_loss: 1.0423, g_loss: 1.7016, D(x): 0.60, D(G(z)): 0.26
Epoch [113/200], Step [0/1875], d_loss: 1.1020, g_loss: 1.0794, D(x): 0.70, D(G(z)): 0.44
Epoch [113/200], Step [1000/1875], d_loss: 1.1647, g_loss: 2.0496, D(x): 0.54, D(G(z)): 0.26
Epoch [114/200], Step [0/1875], d_loss: 1.1799, g_loss: 1.5188, D(x): 0.64, D(G(z)): 0.39
Epoch [114/200], Step [1000/1875], d_loss: 1.0539, g_loss: 1.3321, D(x): 0.72, D(G(z)): 0.40
Epoch [115/200], Step [0/1875], d_loss: 0.9181, g_loss: 1.3867, D(x): 0.71, D(G(z)): 0.35
Epoch [115/200], Step [1000/1875], d_loss: 1.0679, g_loss: 1.9318, D(x): 0.59, D(G(z)): 0.26
Epoch [116/200], Step [0/1875], d_loss: 1.0790, g_loss: 1.1137, D(x): 0.64, D(G(z)): 0.36
Epoch [116/200], Step [1000/1875], d_loss: 1.2793, g_loss: 1.0888, D(x): 0.73, D(G(z)): 0.48
Epoch [117/200], Step [0/1875], d_loss: 0.9659, g_loss: 1.6854, D(x): 0.63, D(G(z)): 0.26
Epoch [117/200], Step [1000/1875], d_loss: 1.0517, g_loss: 1.1859, D(x): 0.68, D(G(z)): 0.38
Epoch [118/200], Step [0/1875], d_loss: 1.0606, g_loss: 1.4192, D(x): 0.67, D(G(z)): 0.29
Epoch [118/200], Step [1000/1875], d_loss: 1.0837, g_loss: 1.5058, D(x): 0.61, D(G(z)): 0.32
Epoch [119/200], Step [0/1875], d_loss: 0.9450, g_loss: 1.2550, D(x): 0.71, D(G(z)): 0.35
Epoch [119/200], Step [1000/1875], d_loss: 1.1078, g_loss: 1.7936, D(x): 0.55, D(G(z)): 0.25
Epoch [120/200], Step [0/1875], d_loss: 0.9814, g_loss: 1.1776, D(x): 0.69, D(G(z)): 0.35
Epoch [120/200], Step [1000/1875], d_loss: 1.0611, g_loss: 1.3892, D(x): 0.59, D(G(z)): 0.31
Epoch [121/200], Step [0/1875], d_loss: 0.9461, g_loss: 1.2199, D(x): 0.70, D(G(z)): 0.36
Epoch [121/200], Step [1000/1875], d_loss: 0.9500, g_loss: 1.2922, D(x): 0.62, D(G(z)): 0.28
Epoch [122/200], Step [0/1875], d_loss: 0.8209, g_loss: 1.4023, D(x): 0.76, D(G(z)): 0.32
Epoch [122/200], Step [1000/1875], d_loss: 1.0864, g_loss: 1.0152, D(x): 0.59, D(G(z)): 0.32
Epoch [123/200], Step [0/1875], d_loss: 1.1689, g_loss: 1.4938, D(x): 0.59, D(G(z)): 0.27
Epoch [123/200], Step [1000/1875], d_loss: 1.0686, g_loss: 1.1028, D(x): 0.64, D(G(z)): 0.33
Epoch [124/200], Step [0/1875], d_loss: 0.9185, g_loss: 1.1483, D(x): 0.72, D(G(z)): 0.33
Epoch [124/200], Step [1000/1875], d_loss: 1.0521, g_loss: 1.0809, D(x): 0.64, D(G(z)): 0.30
Epoch [125/200], Step [0/1875], d_loss: 1.0460, g_loss: 1.7116, D(x): 0.63, D(G(z)): 0.32
Epoch [125/200], Step [1000/1875], d_loss: 1.2099, g_loss: 1.4824, D(x): 0.64, D(G(z)): 0.35
Epoch [126/200], Step [0/1875], d_loss: 1.0053, g_loss: 1.1960, D(x): 0.69, D(G(z)): 0.36
Epoch [126/200], Step [1000/1875], d_loss: 0.9684, g_loss: 1.1075, D(x): 0.66, D(G(z)): 0.34
Epoch [127/200], Step [0/1875], d_loss: 0.7114, g_loss: 1.2725, D(x): 0.76, D(G(z)): 0.30
Epoch [127/200], Step [1000/1875], d_loss: 0.8682, g_loss: 1.3727, D(x): 0.63, D(G(z)): 0.26
Epoch [128/200], Step [0/1875], d_loss: 0.9651, g_loss: 1.1287, D(x): 0.74, D(G(z)): 0.39
Epoch [128/200], Step [1000/1875], d_loss: 0.7600, g_loss: 1.4872, D(x): 0.73, D(G(z)): 0.30
Epoch [129/200], Step [0/1875], d_loss: 1.0353, g_loss: 1.1982, D(x): 0.73, D(G(z)): 0.38
Epoch [129/200], Step [1000/1875], d_loss: 0.9312, g_loss: 1.6565, D(x): 0.67, D(G(z)): 0.30
Epoch [130/200], Step [0/1875], d_loss: 0.7257, g_loss: 1.1873, D(x): 0.69, D(G(z)): 0.22
Epoch [130/200], Step [1000/1875], d_loss: 0.8490, g_loss: 1.5466, D(x): 0.65, D(G(z)): 0.25
Epoch [131/200], Step [0/1875], d_loss: 0.8980, g_loss: 1.5924, D(x): 0.68, D(G(z)): 0.28
Epoch [131/200], Step [1000/1875], d_loss: 0.9562, g_loss: 1.5058, D(x): 0.72, D(G(z)): 0.36
Epoch [132/200], Step [0/1875], d_loss: 1.0407, g_loss: 1.7313, D(x): 0.59, D(G(z)): 0.28
Epoch [132/200], Step [1000/1875], d_loss: 0.8018, g_loss: 1.4991, D(x): 0.72, D(G(z)): 0.31
Epoch [133/200], Step [0/1875], d_loss: 1.0846, g_loss: 1.0952, D(x): 0.69, D(G(z)): 0.38
Epoch [133/200], Step [1000/1875], d_loss: 0.8227, g_loss: 1.0884, D(x): 0.73, D(G(z)): 0.32
Epoch [134/200], Step [0/1875], d_loss: 0.9787, g_loss: 1.4190, D(x): 0.71, D(G(z)): 0.36
Epoch [134/200], Step [1000/1875], d_loss: 1.0852, g_loss: 1.8930, D(x): 0.60, D(G(z)): 0.26
Epoch [135/200], Step [0/1875], d_loss: 1.1340, g_loss: 1.4754, D(x): 0.53, D(G(z)): 0.26
Epoch [135/200], Step [1000/1875], d_loss: 0.8791, g_loss: 1.6002, D(x): 0.73, D(G(z)): 0.34
Epoch [136/200], Step [0/1875], d_loss: 0.9289, g_loss: 1.2938, D(x): 0.74, D(G(z)): 0.39
Epoch [136/200], Step [1000/1875], d_loss: 0.8836, g_loss: 1.4500, D(x): 0.67, D(G(z)): 0.27
Epoch [137/200], Step [0/1875], d_loss: 0.9663, g_loss: 1.2554, D(x): 0.72, D(G(z)): 0.32
Epoch [137/200], Step [1000/1875], d_loss: 0.7621, g_loss: 1.5249, D(x): 0.70, D(G(z)): 0.26
Epoch [138/200], Step [0/1875], d_loss: 1.1226, g_loss: 1.3983, D(x): 0.62, D(G(z)): 0.37
Epoch [138/200], Step [1000/1875], d_loss: 0.8552, g_loss: 1.2803, D(x): 0.68, D(G(z)): 0.30
Epoch [139/200], Step [0/1875], d_loss: 1.1601, g_loss: 1.5479, D(x): 0.66, D(G(z)): 0.37
Epoch [139/200], Step [1000/1875], d_loss: 0.9467, g_loss: 1.3331, D(x): 0.78, D(G(z)): 0.41
Epoch [140/200], Step [0/1875], d_loss: 0.9792, g_loss: 1.6201, D(x): 0.72, D(G(z)): 0.33
Epoch [140/200], Step [1000/1875], d_loss: 1.0290, g_loss: 1.6335, D(x): 0.69, D(G(z)): 0.36
Epoch [141/200], Step [0/1875], d_loss: 0.8760, g_loss: 1.4903, D(x): 0.73, D(G(z)): 0.32
Epoch [141/200], Step [1000/1875], d_loss: 1.1730, g_loss: 1.3827, D(x): 0.65, D(G(z)): 0.35
Epoch [142/200], Step [0/1875], d_loss: 1.2059, g_loss: 1.6793, D(x): 0.58, D(G(z)): 0.32
Epoch [142/200], Step [1000/1875], d_loss: 1.0551, g_loss: 1.3108, D(x): 0.64, D(G(z)): 0.36
Epoch [143/200], Step [0/1875], d_loss: 1.1493, g_loss: 1.2051, D(x): 0.76, D(G(z)): 0.49
Epoch [143/200], Step [1000/1875], d_loss: 0.9853, g_loss: 0.9375, D(x): 0.69, D(G(z)): 0.33
Epoch [144/200], Step [0/1875], d_loss: 0.9930, g_loss: 1.2293, D(x): 0.72, D(G(z)): 0.39
Epoch [144/200], Step [1000/1875], d_loss: 1.1021, g_loss: 1.4031, D(x): 0.58, D(G(z)): 0.31
Epoch [145/200], Step [0/1875], d_loss: 1.0788, g_loss: 1.1353, D(x): 0.76, D(G(z)): 0.45
Epoch [145/200], Step [1000/1875], d_loss: 0.9493, g_loss: 1.9824, D(x): 0.66, D(G(z)): 0.28
Epoch [146/200], Step [0/1875], d_loss: 0.9586, g_loss: 1.3733, D(x): 0.62, D(G(z)): 0.24
Epoch [146/200], Step [1000/1875], d_loss: 1.1211, g_loss: 1.1700, D(x): 0.66, D(G(z)): 0.35
Epoch [147/200], Step [0/1875], d_loss: 1.1163, g_loss: 1.6816, D(x): 0.60, D(G(z)): 0.31
Epoch [147/200], Step [1000/1875], d_loss: 1.1130, g_loss: 1.5149, D(x): 0.61, D(G(z)): 0.33
Epoch [148/200], Step [0/1875], d_loss: 0.9912, g_loss: 1.5274, D(x): 0.65, D(G(z)): 0.30
Epoch [148/200], Step [1000/1875], d_loss: 0.7933, g_loss: 1.3911, D(x): 0.73, D(G(z)): 0.30
Epoch [149/200], Step [0/1875], d_loss: 0.7205, g_loss: 1.7141, D(x): 0.75, D(G(z)): 0.25
Epoch [149/200], Step [1000/1875], d_loss: 1.0681, g_loss: 1.2448, D(x): 0.74, D(G(z)): 0.42
Epoch [150/200], Step [0/1875], d_loss: 0.7419, g_loss: 1.4390, D(x): 0.68, D(G(z)): 0.23
Epoch [150/200], Step [1000/1875], d_loss: 1.0537, g_loss: 1.4104, D(x): 0.65, D(G(z)): 0.33
Epoch [151/200], Step [0/1875], d_loss: 0.7947, g_loss: 1.2123, D(x): 0.72, D(G(z)): 0.31
Epoch [151/200], Step [1000/1875], d_loss: 0.9032, g_loss: 1.7344, D(x): 0.58, D(G(z)): 0.18
Epoch [152/200], Step [0/1875], d_loss: 0.9012, g_loss: 1.8501, D(x): 0.67, D(G(z)): 0.25
Epoch [152/200], Step [1000/1875], d_loss: 0.9152, g_loss: 1.6020, D(x): 0.65, D(G(z)): 0.28
Epoch [153/200], Step [0/1875], d_loss: 1.1215, g_loss: 1.6962, D(x): 0.59, D(G(z)): 0.28
Epoch [153/200], Step [1000/1875], d_loss: 0.9356, g_loss: 1.3529, D(x): 0.75, D(G(z)): 0.37
Epoch [154/200], Step [0/1875], d_loss: 0.9896, g_loss: 1.3403, D(x): 0.74, D(G(z)): 0.37
Epoch [154/200], Step [1000/1875], d_loss: 1.0119, g_loss: 1.2061, D(x): 0.74, D(G(z)): 0.42
Epoch [155/200], Step [0/1875], d_loss: 1.0402, g_loss: 1.2992, D(x): 0.67, D(G(z)): 0.35
Epoch [155/200], Step [1000/1875], d_loss: 0.9205, g_loss: 1.7426, D(x): 0.68, D(G(z)): 0.32
Epoch [156/200], Step [0/1875], d_loss: 0.9372, g_loss: 0.8833, D(x): 0.73, D(G(z)): 0.37
Epoch [156/200], Step [1000/1875], d_loss: 1.2032, g_loss: 1.1325, D(x): 0.61, D(G(z)): 0.34
Epoch [157/200], Step [0/1875], d_loss: 0.9232, g_loss: 1.3139, D(x): 0.71, D(G(z)): 0.36
Epoch [157/200], Step [1000/1875], d_loss: 1.0662, g_loss: 1.0879, D(x): 0.77, D(G(z)): 0.45
Epoch [158/200], Step [0/1875], d_loss: 1.0168, g_loss: 1.1149, D(x): 0.63, D(G(z)): 0.32
Epoch [158/200], Step [1000/1875], d_loss: 0.8170, g_loss: 1.6005, D(x): 0.73, D(G(z)): 0.29
Epoch [159/200], Step [0/1875], d_loss: 0.9503, g_loss: 0.9681, D(x): 0.75, D(G(z)): 0.38
Epoch [159/200], Step [1000/1875], d_loss: 1.0097, g_loss: 1.1410, D(x): 0.69, D(G(z)): 0.38
Epoch [160/200], Step [0/1875], d_loss: 0.8961, g_loss: 1.3045, D(x): 0.78, D(G(z)): 0.42
Epoch [160/200], Step [1000/1875], d_loss: 0.8125, g_loss: 1.5028, D(x): 0.73, D(G(z)): 0.31
Epoch [161/200], Step [0/1875], d_loss: 0.9205, g_loss: 1.6392, D(x): 0.68, D(G(z)): 0.27
Epoch [161/200], Step [1000/1875], d_loss: 0.8256, g_loss: 1.3770, D(x): 0.72, D(G(z)): 0.32
Epoch [162/200], Step [0/1875], d_loss: 1.0830, g_loss: 1.5884, D(x): 0.61, D(G(z)): 0.36
Epoch [162/200], Step [1000/1875], d_loss: 0.9695, g_loss: 1.7384, D(x): 0.63, D(G(z)): 0.27
Epoch [163/200], Step [0/1875], d_loss: 1.0718, g_loss: 1.7019, D(x): 0.71, D(G(z)): 0.35
Epoch [163/200], Step [1000/1875], d_loss: 0.7365, g_loss: 1.5347, D(x): 0.72, D(G(z)): 0.28
Epoch [164/200], Step [0/1875], d_loss: 1.0448, g_loss: 1.4188, D(x): 0.56, D(G(z)): 0.25
Epoch [164/200], Step [1000/1875], d_loss: 0.7024, g_loss: 1.1493, D(x): 0.80, D(G(z)): 0.34
Epoch [165/200], Step [0/1875], d_loss: 0.6509, g_loss: 1.3696, D(x): 0.80, D(G(z)): 0.28
Epoch [165/200], Step [1000/1875], d_loss: 0.9247, g_loss: 1.3655, D(x): 0.70, D(G(z)): 0.35
Epoch [166/200], Step [0/1875], d_loss: 0.9052, g_loss: 1.0978, D(x): 0.76, D(G(z)): 0.39
Epoch [166/200], Step [1000/1875], d_loss: 0.7881, g_loss: 1.3715, D(x): 0.73, D(G(z)): 0.29
Epoch [167/200], Step [0/1875], d_loss: 1.0630, g_loss: 1.3438, D(x): 0.70, D(G(z)): 0.40
Epoch [167/200], Step [1000/1875], d_loss: 1.1506, g_loss: 1.6357, D(x): 0.54, D(G(z)): 0.24
Epoch [168/200], Step [0/1875], d_loss: 0.7992, g_loss: 1.6564, D(x): 0.73, D(G(z)): 0.25
Epoch [168/200], Step [1000/1875], d_loss: 0.9592, g_loss: 1.4136, D(x): 0.65, D(G(z)): 0.30
Epoch [169/200], Step [0/1875], d_loss: 0.9599, g_loss: 1.2311, D(x): 0.66, D(G(z)): 0.31
Epoch [169/200], Step [1000/1875], d_loss: 0.9924, g_loss: 1.6626, D(x): 0.68, D(G(z)): 0.34
Epoch [170/200], Step [0/1875], d_loss: 0.8713, g_loss: 1.8207, D(x): 0.80, D(G(z)): 0.40
Epoch [170/200], Step [1000/1875], d_loss: 1.1869, g_loss: 1.2282, D(x): 0.66, D(G(z)): 0.38
Epoch [171/200], Step [0/1875], d_loss: 1.0279, g_loss: 1.1309, D(x): 0.63, D(G(z)): 0.32
Epoch [171/200], Step [1000/1875], d_loss: 0.8797, g_loss: 2.0040, D(x): 0.71, D(G(z)): 0.30
Epoch [172/200], Step [0/1875], d_loss: 0.9126, g_loss: 1.6225, D(x): 0.65, D(G(z)): 0.24
Epoch [172/200], Step [1000/1875], d_loss: 0.8287, g_loss: 1.5217, D(x): 0.69, D(G(z)): 0.27
Epoch [173/200], Step [0/1875], d_loss: 0.9320, g_loss: 1.1766, D(x): 0.68, D(G(z)): 0.28
Epoch [173/200], Step [1000/1875], d_loss: 1.0596, g_loss: 2.0341, D(x): 0.69, D(G(z)): 0.29
Epoch [174/200], Step [0/1875], d_loss: 0.8312, g_loss: 1.5397, D(x): 0.79, D(G(z)): 0.33
Epoch [174/200], Step [1000/1875], d_loss: 0.7674, g_loss: 1.2422, D(x): 0.70, D(G(z)): 0.24
Epoch [175/200], Step [0/1875], d_loss: 0.7503, g_loss: 1.7233, D(x): 0.73, D(G(z)): 0.24
Epoch [175/200], Step [1000/1875], d_loss: 1.0236, g_loss: 1.3441, D(x): 0.64, D(G(z)): 0.34
Epoch [176/200], Step [0/1875], d_loss: 0.9639, g_loss: 1.4112, D(x): 0.73, D(G(z)): 0.39
Epoch [176/200], Step [1000/1875], d_loss: 0.7873, g_loss: 1.5375, D(x): 0.76, D(G(z)): 0.32
Epoch [177/200], Step [0/1875], d_loss: 0.8955, g_loss: 1.4132, D(x): 0.68, D(G(z)): 0.29
Epoch [177/200], Step [1000/1875], d_loss: 1.1728, g_loss: 1.5841, D(x): 0.60, D(G(z)): 0.32
Epoch [178/200], Step [0/1875], d_loss: 0.8312, g_loss: 1.3275, D(x): 0.72, D(G(z)): 0.27
Epoch [178/200], Step [1000/1875], d_loss: 1.0104, g_loss: 1.3960, D(x): 0.74, D(G(z)): 0.38
Epoch [179/200], Step [0/1875], d_loss: 0.8851, g_loss: 1.2724, D(x): 0.77, D(G(z)): 0.39
Epoch [179/200], Step [1000/1875], d_loss: 1.0904, g_loss: 1.3150, D(x): 0.65, D(G(z)): 0.37
Epoch [180/200], Step [0/1875], d_loss: 0.8384, g_loss: 1.4742, D(x): 0.71, D(G(z)): 0.29
Epoch [180/200], Step [1000/1875], d_loss: 1.2366, g_loss: 1.3583, D(x): 0.69, D(G(z)): 0.46
Epoch [181/200], Step [0/1875], d_loss: 1.1119, g_loss: 1.6124, D(x): 0.72, D(G(z)): 0.37
Epoch [181/200], Step [1000/1875], d_loss: 1.1789, g_loss: 1.1550, D(x): 0.59, D(G(z)): 0.31
Epoch [182/200], Step [0/1875], d_loss: 1.0281, g_loss: 0.8599, D(x): 0.74, D(G(z)): 0.41
Epoch [182/200], Step [1000/1875], d_loss: 1.1698, g_loss: 1.5769, D(x): 0.63, D(G(z)): 0.36
Epoch [183/200], Step [0/1875], d_loss: 0.8879, g_loss: 1.2538, D(x): 0.73, D(G(z)): 0.34
Epoch [183/200], Step [1000/1875], d_loss: 0.9862, g_loss: 1.7838, D(x): 0.61, D(G(z)): 0.28
Epoch [184/200], Step [0/1875], d_loss: 0.8218, g_loss: 1.2922, D(x): 0.67, D(G(z)): 0.25
Epoch [184/200], Step [1000/1875], d_loss: 0.8309, g_loss: 1.4066, D(x): 0.81, D(G(z)): 0.36
Epoch [185/200], Step [0/1875], d_loss: 0.8038, g_loss: 1.1647, D(x): 0.68, D(G(z)): 0.25
Epoch [185/200], Step [1000/1875], d_loss: 1.1130, g_loss: 1.4807, D(x): 0.63, D(G(z)): 0.31
Epoch [186/200], Step [0/1875], d_loss: 1.3574, g_loss: 1.8987, D(x): 0.53, D(G(z)): 0.29
Epoch [186/200], Step [1000/1875], d_loss: 1.0933, g_loss: 1.5170, D(x): 0.59, D(G(z)): 0.29
Epoch [187/200], Step [0/1875], d_loss: 0.9468, g_loss: 1.6168, D(x): 0.72, D(G(z)): 0.35
Epoch [187/200], Step [1000/1875], d_loss: 0.9485, g_loss: 1.8171, D(x): 0.69, D(G(z)): 0.29
Epoch [188/200], Step [0/1875], d_loss: 0.9278, g_loss: 1.0927, D(x): 0.66, D(G(z)): 0.27
Epoch [188/200], Step [1000/1875], d_loss: 0.8841, g_loss: 1.4329, D(x): 0.74, D(G(z)): 0.34
Epoch [189/200], Step [0/1875], d_loss: 1.1402, g_loss: 1.1908, D(x): 0.60, D(G(z)): 0.34
Epoch [189/200], Step [1000/1875], d_loss: 0.9516, g_loss: 1.2448, D(x): 0.71, D(G(z)): 0.35
Epoch [190/200], Step [0/1875], d_loss: 0.8385, g_loss: 1.1752, D(x): 0.77, D(G(z)): 0.32
Epoch [190/200], Step [1000/1875], d_loss: 1.1430, g_loss: 1.3168, D(x): 0.70, D(G(z)): 0.41
Epoch [191/200], Step [0/1875], d_loss: 0.8999, g_loss: 1.4734, D(x): 0.65, D(G(z)): 0.25
Epoch [191/200], Step [1000/1875], d_loss: 0.8006, g_loss: 1.3487, D(x): 0.68, D(G(z)): 0.20
Epoch [192/200], Step [0/1875], d_loss: 0.9304, g_loss: 1.5564, D(x): 0.71, D(G(z)): 0.32
Epoch [192/200], Step [1000/1875], d_loss: 0.8803, g_loss: 1.3595, D(x): 0.62, D(G(z)): 0.23
Epoch [193/200], Step [0/1875], d_loss: 0.8008, g_loss: 1.4470, D(x): 0.67, D(G(z)): 0.24
Epoch [193/200], Step [1000/1875], d_loss: 1.0546, g_loss: 1.0424, D(x): 0.83, D(G(z)): 0.47
Epoch [194/200], Step [0/1875], d_loss: 0.8909, g_loss: 2.0232, D(x): 0.65, D(G(z)): 0.25
Epoch [194/200], Step [1000/1875], d_loss: 1.1190, g_loss: 1.4303, D(x): 0.60, D(G(z)): 0.28
Epoch [195/200], Step [0/1875], d_loss: 0.9886, g_loss: 1.5795, D(x): 0.59, D(G(z)): 0.27
Epoch [195/200], Step [1000/1875], d_loss: 0.8679, g_loss: 1.7040, D(x): 0.67, D(G(z)): 0.24
Epoch [196/200], Step [0/1875], d_loss: 1.0795, g_loss: 1.9095, D(x): 0.62, D(G(z)): 0.29
Epoch [196/200], Step [1000/1875], d_loss: 0.9541, g_loss: 1.3313, D(x): 0.72, D(G(z)): 0.33
Epoch [197/200], Step [0/1875], d_loss: 1.0868, g_loss: 1.6374, D(x): 0.78, D(G(z)): 0.45
Epoch [197/200], Step [1000/1875], d_loss: 1.0295, g_loss: 1.5136, D(x): 0.67, D(G(z)): 0.36
Epoch [198/200], Step [0/1875], d_loss: 1.0157, g_loss: 1.9654, D(x): 0.56, D(G(z)): 0.17
Epoch [198/200], Step [1000/1875], d_loss: 0.7938, g_loss: 1.6147, D(x): 0.79, D(G(z)): 0.31
Epoch [199/200], Step [0/1875], d_loss: 0.7751, g_loss: 1.2779, D(x): 0.84, D(G(z)): 0.39
Epoch [199/200], Step [1000/1875], d_loss: 0.9865, g_loss: 1.4814, D(x): 0.65, D(G(z)): 0.27

fake images

1
2
3

z = torch.randn(1, latent_size).to(device)
fake_images = G(z).view(28, 28).data.cpu().numpy()
plt.imshow(fake_images)

<matplotlib.image.AxesImage at 0x7f55b00136d8>

png

真实图片

1	plt.imshow(images[0].view(28,28).data.cpu().numpy())

<matplotlib.image.AxesImage at 0x7f55b09e7f60>

png

DCGAN

UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS

图片下载地址
 https://drive.google.com/drive/folders/0B7EVK8r0v71pbWNEUjJKdDQ3dGc

1	import torchvision.utils as vutils

1	# !ls celeba/img_align_celeba/img_align_celeba_png

image_size=64
batch_size=128
dataroot="celeba/img_align_celeba"
num_workers = 2
dataset = torchvision.datasets.ImageFolder(root=dataroot, transform=transforms.Compose([
    transforms.Resize(image_size),
    transforms.CenterCrop(image_size),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
]))
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=num_workers)

real_batch=next(iter(dataloader))
plt.figure(figsize=(8,8))
plt.axis=("off")
plt.title("Training Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=2, normalize=True).cpu(), (1,2,0)))

<matplotlib.image.AxesImage at 0x7f6db16dafd0>

png

我们把模型的所有参数都初始化城mean=0, std=0.2

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

dcgan

nz = 100 # latent vector的大小
ngf = 64 # generator feature map size
ndf = 64 # discriminator feature map size
nc = 3 # color channels

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            # torch.nn.ConvTranspose2d(in_channels, out_channels, 
            # kernel_size, stride=1, padding=0, output_padding=0, groups=1, bias=True, dilation=1)
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)

# Now, we can instantiate the generator and apply the weights_init function. Check out the printed model to see how the generator object is structured.

# Create the generator
netG = Generator().to(device)

# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netG.apply(weights_init)

# Print the model
print(netG)

Generator(
  (main): Sequential(
    (0): ConvTranspose2d(100, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (2): ReLU(inplace)
    (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace)
    (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (8): ReLU(inplace)
    (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (11): ReLU(inplace)
    (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (13): Tanh()
  )
)

Discriminator

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.main = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*8) x 4 x 4
            nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.main(input)

# Now, as with the generator, we can create the discriminator, apply the weights_init function, and print the model’s structure.

# Create the Discriminator
netD = Discriminator().to(device)


# Apply the weights_init function to randomly initialize all weights
#  to mean=0, stdev=0.2.
netD.apply(weights_init)

# Print the model
print(netD)

Discriminator(
  (main): Sequential(
    (0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (1): LeakyReLU(negative_slope=0.2, inplace)
    (2): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (4): LeakyReLU(negative_slope=0.2, inplace)
    (5): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (6): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (7): LeakyReLU(negative_slope=0.2, inplace)
    (8): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (9): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (10): LeakyReLU(negative_slope=0.2, inplace)
    (11): Conv2d(512, 1, kernel_size=(4, 4), stride=(1, 1), bias=False)
    (12): Sigmoid()
  )
)

开始训练

lr = 0.0002
beta1 = 0.5

loss_fn = nn.BCELoss()
fixed_noise = torch.randn(64, nz, 1, 1, device=device)
d_optimizer = torch.optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
g_optimizer = torch.optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

num_epochs = 5
G_losses = []
D_losses = []
for epoch in range(num_epochs):
    for i, data in enumerate(dataloader):
        # 训练discriminator, maximize log(D(x)) + log(1-D(G(z)))
        
        # 首先训练真实图片
        netD.zero_grad()
        
        real_images = data[0].to(device)
        b_size = real_images.size(0)
        label = torch.ones(b_size).to(device)
        output = netD(real_images).view(-1)
        
        
        real_loss = loss_fn(output, label)
        real_loss.backward()
        D_x = output.mean().item()
        
        
        # 然后训练生成的假图片
        noise = torch.randn(b_size, nz, 1, 1, device=device)
        fake_images = netG(noise)
        label.fill_(0)
        output = netD(fake_images.detach()).view(-1)
        fake_loss = loss_fn(output, label)
        fake_loss.backward()
        D_G_z1 = output.mean().item()
        loss_D = real_loss + fake_loss
        d_optimizer.step()
        
        # 训练Generator 
        netG.zero_grad()
        label.fill_(1)
        output = netD(fake_images).view(-1)
        loss_G = loss_fn(output, label)
        loss_G.backward()
        D_G_z2 = output.mean().item()
        g_optimizer.step()
        
        if i % 50 == 0:
            print("[{}/{}] [{}/{}] Loss_D: {:.4f} Loss_G {:.4f} D(x): {:.4f} D(G(z)): {:.4f}/{:.4f}"
                 .format(epoch, num_epochs, i, len(dataloader), loss_D.item(), loss_G.item(), D_x, D_G_z1, D_G_z2))
        
        G_losses.append(loss_G.item())
        D_losses.append(loss_D.item())

[0/5] [0/1583] Loss_D: 1.7977 Loss_G 2.8596 D(x): 0.3357 D(G(z)): 0.3494/0.0786
[0/5] [50/1583] Loss_D: 0.4748 Loss_G 30.1861 D(x): 0.7715 D(G(z)): 0.0000/0.0000
[0/5] [100/1583] Loss_D: 0.1432 Loss_G 8.7877 D(x): 0.9865 D(G(z)): 0.1092/0.0016
[0/5] [150/1583] Loss_D: 0.5332 Loss_G 6.9773 D(x): 0.8701 D(G(z)): 0.2674/0.0030
[0/5] [200/1583] Loss_D: 1.5008 Loss_G 8.1102 D(x): 0.4722 D(G(z)): 0.0029/0.0011
[0/5] [250/1583] Loss_D: 0.3476 Loss_G 5.5318 D(x): 0.8942 D(G(z)): 0.1540/0.0132
[0/5] [300/1583] Loss_D: 0.6494 Loss_G 5.9788 D(x): 0.9072 D(G(z)): 0.3348/0.0124
[0/5] [350/1583] Loss_D: 0.8482 Loss_G 5.6696 D(x): 0.8947 D(G(z)): 0.4554/0.0091
[0/5] [400/1583] Loss_D: 0.5689 Loss_G 3.3358 D(x): 0.7856 D(G(z)): 0.1807/0.0647
[0/5] [450/1583] Loss_D: 0.8698 Loss_G 7.5017 D(x): 0.8675 D(G(z)): 0.4281/0.0022
[0/5] [500/1583] Loss_D: 0.3542 Loss_G 3.1888 D(x): 0.8573 D(G(z)): 0.1214/0.0587
[0/5] [550/1583] Loss_D: 0.3387 Loss_G 3.9772 D(x): 0.7958 D(G(z)): 0.0605/0.0351
[0/5] [600/1583] Loss_D: 0.6330 Loss_G 4.3450 D(x): 0.7693 D(G(z)): 0.1875/0.0238
[0/5] [650/1583] Loss_D: 0.6735 Loss_G 4.8144 D(x): 0.6305 D(G(z)): 0.0358/0.0166
[0/5] [700/1583] Loss_D: 0.3484 Loss_G 4.6406 D(x): 0.8652 D(G(z)): 0.1372/0.0182
[0/5] [750/1583] Loss_D: 0.5287 Loss_G 5.8325 D(x): 0.8684 D(G(z)): 0.2675/0.0056
[0/5] [800/1583] Loss_D: 0.6363 Loss_G 3.1169 D(x): 0.6298 D(G(z)): 0.0332/0.0755
[0/5] [850/1583] Loss_D: 0.4994 Loss_G 5.3602 D(x): 0.8846 D(G(z)): 0.2461/0.0114
[0/5] [900/1583] Loss_D: 0.5199 Loss_G 5.4862 D(x): 0.9498 D(G(z)): 0.2993/0.0118
[0/5] [950/1583] Loss_D: 0.3113 Loss_G 3.8929 D(x): 0.8070 D(G(z)): 0.0317/0.0357
[0/5] [1000/1583] Loss_D: 1.3229 Loss_G 1.8840 D(x): 0.3859 D(G(z)): 0.0013/0.2331
[0/5] [1050/1583] Loss_D: 0.3150 Loss_G 3.5746 D(x): 0.8395 D(G(z)): 0.0970/0.0547
[0/5] [1100/1583] Loss_D: 0.5306 Loss_G 3.1867 D(x): 0.6945 D(G(z)): 0.0447/0.0750
[0/5] [1150/1583] Loss_D: 0.5492 Loss_G 2.5496 D(x): 0.6916 D(G(z)): 0.0663/0.1255
[0/5] [1200/1583] Loss_D: 0.3651 Loss_G 4.2102 D(x): 0.7647 D(G(z)): 0.0440/0.0365
[0/5] [1250/1583] Loss_D: 1.3114 Loss_G 2.9933 D(x): 0.4186 D(G(z)): 0.0093/0.0944
[0/5] [1300/1583] Loss_D: 0.7040 Loss_G 6.9100 D(x): 0.8776 D(G(z)): 0.3483/0.0018
[0/5] [1350/1583] Loss_D: 0.6155 Loss_G 2.0302 D(x): 0.6897 D(G(z)): 0.1118/0.1726
[0/5] [1400/1583] Loss_D: 0.5944 Loss_G 3.1167 D(x): 0.7538 D(G(z)): 0.1957/0.0642
[0/5] [1450/1583] Loss_D: 0.3558 Loss_G 3.7467 D(x): 0.8731 D(G(z)): 0.1555/0.0415
[0/5] [1500/1583] Loss_D: 0.4071 Loss_G 4.1953 D(x): 0.8410 D(G(z)): 0.1453/0.0310
[0/5] [1550/1583] Loss_D: 1.6558 Loss_G 9.1945 D(x): 0.9677 D(G(z)): 0.7053/0.0004
[1/5] [0/1583] Loss_D: 0.5024 Loss_G 4.3460 D(x): 0.8704 D(G(z)): 0.2554/0.0201
[1/5] [50/1583] Loss_D: 0.7825 Loss_G 5.5473 D(x): 0.9305 D(G(z)): 0.4510/0.0072
[1/5] [100/1583] Loss_D: 0.5763 Loss_G 4.2330 D(x): 0.8332 D(G(z)): 0.2738/0.0248
[1/5] [150/1583] Loss_D: 0.5093 Loss_G 3.9376 D(x): 0.8285 D(G(z)): 0.2162/0.0325
[1/5] [200/1583] Loss_D: 0.7584 Loss_G 4.4998 D(x): 0.8351 D(G(z)): 0.3689/0.0258
[1/5] [250/1583] Loss_D: 0.4091 Loss_G 3.9546 D(x): 0.7356 D(G(z)): 0.0257/0.0337
[1/5] [300/1583] Loss_D: 0.5199 Loss_G 4.4009 D(x): 0.8562 D(G(z)): 0.2620/0.0212
[1/5] [350/1583] Loss_D: 1.6999 Loss_G 1.1305 D(x): 0.3153 D(G(z)): 0.0194/0.3955
[1/5] [400/1583] Loss_D: 0.4612 Loss_G 5.0442 D(x): 0.9210 D(G(z)): 0.2755/0.0113
[1/5] [450/1583] Loss_D: 0.3626 Loss_G 2.7311 D(x): 0.8119 D(G(z)): 0.1034/0.1106
[1/5] [500/1583] Loss_D: 0.5614 Loss_G 3.6350 D(x): 0.7820 D(G(z)): 0.1946/0.0512
[1/5] [550/1583] Loss_D: 0.3365 Loss_G 3.3296 D(x): 0.8540 D(G(z)): 0.1276/0.0561
[1/5] [600/1583] Loss_D: 0.9953 Loss_G 1.0561 D(x): 0.4885 D(G(z)): 0.0517/0.4178
[1/5] [650/1583] Loss_D: 0.4633 Loss_G 4.3857 D(x): 0.9219 D(G(z)): 0.2868/0.0181
[1/5] [700/1583] Loss_D: 0.3547 Loss_G 3.1719 D(x): 0.8356 D(G(z)): 0.1229/0.0661
[1/5] [750/1583] Loss_D: 1.4018 Loss_G 7.3128 D(x): 0.9540 D(G(z)): 0.6648/0.0022
[1/5] [800/1583] Loss_D: 1.9716 Loss_G 2.3110 D(x): 0.2525 D(G(z)): 0.0097/0.1644
[1/5] [850/1583] Loss_D: 0.3039 Loss_G 3.3825 D(x): 0.8757 D(G(z)): 0.1389/0.0494
[1/5] [900/1583] Loss_D: 0.4306 Loss_G 4.5716 D(x): 0.9128 D(G(z)): 0.2424/0.0176
[1/5] [950/1583] Loss_D: 1.0529 Loss_G 6.2549 D(x): 0.9377 D(G(z)): 0.5375/0.0043
[1/5] [1000/1583] Loss_D: 0.5825 Loss_G 2.5413 D(x): 0.7108 D(G(z)): 0.1435/0.1155
[1/5] [1050/1583] Loss_D: 0.6516 Loss_G 4.6775 D(x): 0.9519 D(G(z)): 0.4014/0.0170
[1/5] [1100/1583] Loss_D: 0.8078 Loss_G 5.3468 D(x): 0.8942 D(G(z)): 0.4513/0.0077
[1/5] [1150/1583] Loss_D: 0.7372 Loss_G 4.2160 D(x): 0.8662 D(G(z)): 0.3771/0.0272
[1/5] [1200/1583] Loss_D: 0.5704 Loss_G 1.7837 D(x): 0.6827 D(G(z)): 0.0922/0.2175
[1/5] [1250/1583] Loss_D: 0.8721 Loss_G 4.8623 D(x): 0.9443 D(G(z)): 0.4977/0.0137
[1/5] [1300/1583] Loss_D: 0.5091 Loss_G 2.4733 D(x): 0.6754 D(G(z)): 0.0485/0.1242
[1/5] [1350/1583] Loss_D: 0.4865 Loss_G 3.0695 D(x): 0.8064 D(G(z)): 0.1955/0.0689
[1/5] [1400/1583] Loss_D: 0.6490 Loss_G 4.3856 D(x): 0.9040 D(G(z)): 0.3590/0.0200
[1/5] [1450/1583] Loss_D: 0.6000 Loss_G 2.2117 D(x): 0.7705 D(G(z)): 0.2435/0.1419
[1/5] [1500/1583] Loss_D: 0.5049 Loss_G 3.4771 D(x): 0.8402 D(G(z)): 0.2365/0.0487
[1/5] [1550/1583] Loss_D: 0.5885 Loss_G 1.5197 D(x): 0.6468 D(G(z)): 0.0694/0.2862
[2/5] [0/1583] Loss_D: 0.5091 Loss_G 2.2415 D(x): 0.7458 D(G(z)): 0.1528/0.1331
[2/5] [50/1583] Loss_D: 0.4685 Loss_G 2.8283 D(x): 0.8897 D(G(z)): 0.2576/0.0899
[2/5] [100/1583] Loss_D: 0.5364 Loss_G 2.2865 D(x): 0.7544 D(G(z)): 0.1845/0.1296
[2/5] [150/1583] Loss_D: 2.4751 Loss_G 4.7502 D(x): 0.9278 D(G(z)): 0.8115/0.0218
[2/5] [200/1583] Loss_D: 1.7663 Loss_G 1.6306 D(x): 0.2388 D(G(z)): 0.0119/0.2518
[2/5] [250/1583] Loss_D: 0.6184 Loss_G 1.8157 D(x): 0.6371 D(G(z)): 0.0831/0.2129
[2/5] [300/1583] Loss_D: 0.6009 Loss_G 2.4621 D(x): 0.6639 D(G(z)): 0.0986/0.1299
[2/5] [350/1583] Loss_D: 0.6172 Loss_G 2.7100 D(x): 0.7548 D(G(z)): 0.2272/0.0928
[2/5] [400/1583] Loss_D: 0.5001 Loss_G 2.0378 D(x): 0.6971 D(G(z)): 0.0869/0.1678
[2/5] [450/1583] Loss_D: 0.6404 Loss_G 3.3460 D(x): 0.8992 D(G(z)): 0.3705/0.0574
[2/5] [500/1583] Loss_D: 0.5403 Loss_G 2.1565 D(x): 0.6950 D(G(z)): 0.1098/0.1509
[2/5] [550/1583] Loss_D: 0.5993 Loss_G 3.6174 D(x): 0.9018 D(G(z)): 0.3564/0.0417
[2/5] [600/1583] Loss_D: 1.0482 Loss_G 3.6277 D(x): 0.9294 D(G(z)): 0.5477/0.0558
[2/5] [650/1583] Loss_D: 0.4903 Loss_G 2.8267 D(x): 0.8284 D(G(z)): 0.2277/0.0809
[2/5] [700/1583] Loss_D: 0.6068 Loss_G 2.0575 D(x): 0.6432 D(G(z)): 0.0900/0.1623
[2/5] [750/1583] Loss_D: 1.4213 Loss_G 1.1597 D(x): 0.3157 D(G(z)): 0.0459/0.3713
[2/5] [800/1583] Loss_D: 0.5707 Loss_G 2.9375 D(x): 0.8297 D(G(z)): 0.2824/0.0749
[2/5] [850/1583] Loss_D: 0.8145 Loss_G 0.8862 D(x): 0.5465 D(G(z)): 0.0760/0.4527
[2/5] [900/1583] Loss_D: 0.7114 Loss_G 1.6350 D(x): 0.6121 D(G(z)): 0.1375/0.2354
[2/5] [950/1583] Loss_D: 0.6885 Loss_G 2.2735 D(x): 0.6473 D(G(z)): 0.1660/0.1418
[2/5] [1000/1583] Loss_D: 1.0785 Loss_G 1.0148 D(x): 0.4366 D(G(z)): 0.0730/0.4223
[2/5] [1050/1583] Loss_D: 0.7579 Loss_G 2.5076 D(x): 0.6528 D(G(z)): 0.1961/0.1142
[2/5] [1100/1583] Loss_D: 0.6557 Loss_G 3.5820 D(x): 0.8655 D(G(z)): 0.3631/0.0372
[2/5] [1150/1583] Loss_D: 0.6402 Loss_G 3.3918 D(x): 0.9224 D(G(z)): 0.3697/0.0502
[2/5] [1200/1583] Loss_D: 0.6989 Loss_G 2.1415 D(x): 0.6174 D(G(z)): 0.1267/0.1470
[2/5] [1250/1583] Loss_D: 0.6699 Loss_G 1.9413 D(x): 0.6219 D(G(z)): 0.0995/0.1874
[2/5] [1300/1583] Loss_D: 0.6479 Loss_G 1.8731 D(x): 0.6013 D(G(z)): 0.0677/0.1886
[2/5] [1350/1583] Loss_D: 0.5023 Loss_G 3.0319 D(x): 0.8700 D(G(z)): 0.2779/0.0599
[2/5] [1400/1583] Loss_D: 0.4328 Loss_G 2.8918 D(x): 0.7801 D(G(z)): 0.1382/0.0732
[2/5] [1450/1583] Loss_D: 0.6579 Loss_G 2.0119 D(x): 0.7162 D(G(z)): 0.2334/0.1683
[2/5] [1500/1583] Loss_D: 0.8299 Loss_G 3.7579 D(x): 0.8492 D(G(z)): 0.4391/0.0331
[2/5] [1550/1583] Loss_D: 0.6887 Loss_G 1.4614 D(x): 0.6397 D(G(z)): 0.1633/0.2818
[3/5] [0/1583] Loss_D: 0.8251 Loss_G 2.7054 D(x): 0.7448 D(G(z)): 0.3643/0.0842
[3/5] [50/1583] Loss_D: 0.6720 Loss_G 2.8488 D(x): 0.8670 D(G(z)): 0.3652/0.0798
[3/5] [100/1583] Loss_D: 0.6498 Loss_G 1.4725 D(x): 0.7225 D(G(z)): 0.2206/0.2821
[3/5] [150/1583] Loss_D: 1.0247 Loss_G 1.1697 D(x): 0.4694 D(G(z)): 0.1067/0.3692
[3/5] [200/1583] Loss_D: 0.5313 Loss_G 2.4117 D(x): 0.8255 D(G(z)): 0.2553/0.1162
[3/5] [250/1583] Loss_D: 0.7865 Loss_G 2.2379 D(x): 0.5887 D(G(z)): 0.1421/0.1469
[3/5] [300/1583] Loss_D: 1.1039 Loss_G 3.4455 D(x): 0.8690 D(G(z)): 0.5555/0.0467
[3/5] [350/1583] Loss_D: 0.5300 Loss_G 1.9104 D(x): 0.7838 D(G(z)): 0.2207/0.1845
[3/5] [400/1583] Loss_D: 0.7535 Loss_G 3.2029 D(x): 0.7946 D(G(z)): 0.3583/0.0539
[3/5] [450/1583] Loss_D: 0.7322 Loss_G 4.1419 D(x): 0.8885 D(G(z)): 0.4089/0.0217
[3/5] [500/1583] Loss_D: 0.5901 Loss_G 2.4395 D(x): 0.7824 D(G(z)): 0.2573/0.1048
[3/5] [550/1583] Loss_D: 0.6639 Loss_G 3.1330 D(x): 0.8085 D(G(z)): 0.3284/0.0604
[3/5] [600/1583] Loss_D: 0.5979 Loss_G 2.5612 D(x): 0.8028 D(G(z)): 0.2748/0.0973
[3/5] [650/1583] Loss_D: 0.6524 Loss_G 2.2008 D(x): 0.7211 D(G(z)): 0.2281/0.1383
[3/5] [700/1583] Loss_D: 0.5078 Loss_G 2.2305 D(x): 0.7849 D(G(z)): 0.1987/0.1305
[3/5] [750/1583] Loss_D: 0.7095 Loss_G 3.5083 D(x): 0.8811 D(G(z)): 0.3953/0.0417
[3/5] [800/1583] Loss_D: 0.7160 Loss_G 2.6990 D(x): 0.8064 D(G(z)): 0.3518/0.0900
[3/5] [850/1583] Loss_D: 0.6407 Loss_G 3.0253 D(x): 0.8553 D(G(z)): 0.3457/0.0606
[3/5] [900/1583] Loss_D: 0.7381 Loss_G 3.8821 D(x): 0.8539 D(G(z)): 0.3712/0.0279
[3/5] [950/1583] Loss_D: 1.0212 Loss_G 1.1013 D(x): 0.5035 D(G(z)): 0.1802/0.3981
[3/5] [1000/1583] Loss_D: 0.5352 Loss_G 2.1082 D(x): 0.7537 D(G(z)): 0.1909/0.1556
[3/5] [1050/1583] Loss_D: 0.9204 Loss_G 1.1990 D(x): 0.5621 D(G(z)): 0.2122/0.3387
[3/5] [1100/1583] Loss_D: 1.3896 Loss_G 3.7979 D(x): 0.8729 D(G(z)): 0.6477/0.0351
[3/5] [1150/1583] Loss_D: 0.6079 Loss_G 2.3365 D(x): 0.7236 D(G(z)): 0.1868/0.1222
[3/5] [1200/1583] Loss_D: 0.7446 Loss_G 3.2400 D(x): 0.8669 D(G(z)): 0.4066/0.0536
[3/5] [1250/1583] Loss_D: 0.5165 Loss_G 2.2988 D(x): 0.7275 D(G(z)): 0.1453/0.1266
[3/5] [1300/1583] Loss_D: 0.4456 Loss_G 2.2971 D(x): 0.7558 D(G(z)): 0.1283/0.1286
[3/5] [1350/1583] Loss_D: 0.6839 Loss_G 1.8744 D(x): 0.7300 D(G(z)): 0.2578/0.1925
[3/5] [1400/1583] Loss_D: 0.5876 Loss_G 3.1330 D(x): 0.8353 D(G(z)): 0.3002/0.0564
[3/5] [1450/1583] Loss_D: 0.5586 Loss_G 3.2172 D(x): 0.9043 D(G(z)): 0.3380/0.0534
[3/5] [1500/1583] Loss_D: 0.5847 Loss_G 2.8399 D(x): 0.8091 D(G(z)): 0.2777/0.0809
[3/5] [1550/1583] Loss_D: 0.4929 Loss_G 2.3813 D(x): 0.7532 D(G(z)): 0.1533/0.1226
[4/5] [0/1583] Loss_D: 0.8560 Loss_G 4.1151 D(x): 0.8905 D(G(z)): 0.4680/0.0250
[4/5] [50/1583] Loss_D: 0.6350 Loss_G 2.4734 D(x): 0.7954 D(G(z)): 0.2928/0.1036
[4/5] [100/1583] Loss_D: 0.5003 Loss_G 2.0825 D(x): 0.7856 D(G(z)): 0.2060/0.1513
[4/5] [150/1583] Loss_D: 0.6394 Loss_G 2.3414 D(x): 0.7299 D(G(z)): 0.2361/0.1241
[4/5] [200/1583] Loss_D: 0.4699 Loss_G 1.9515 D(x): 0.7187 D(G(z)): 0.0963/0.1836
[4/5] [250/1583] Loss_D: 0.6581 Loss_G 1.8691 D(x): 0.6796 D(G(z)): 0.1988/0.1950
[4/5] [300/1583] Loss_D: 0.7072 Loss_G 2.3310 D(x): 0.7996 D(G(z)): 0.3419/0.1218
[4/5] [350/1583] Loss_D: 1.4915 Loss_G 4.6909 D(x): 0.9691 D(G(z)): 0.7055/0.0143
[4/5] [400/1583] Loss_D: 0.7722 Loss_G 3.2458 D(x): 0.8720 D(G(z)): 0.4223/0.0511
[4/5] [450/1583] Loss_D: 1.6807 Loss_G 0.2487 D(x): 0.3045 D(G(z)): 0.2069/0.7933
[4/5] [500/1583] Loss_D: 0.8011 Loss_G 2.0801 D(x): 0.7842 D(G(z)): 0.3814/0.1584
[4/5] [550/1583] Loss_D: 0.7781 Loss_G 1.3220 D(x): 0.5185 D(G(z)): 0.0437/0.3086
[4/5] [600/1583] Loss_D: 0.9146 Loss_G 1.1716 D(x): 0.5058 D(G(z)): 0.0925/0.3569
[4/5] [650/1583] Loss_D: 0.6587 Loss_G 2.8468 D(x): 0.8144 D(G(z)): 0.3266/0.0783
[4/5] [700/1583] Loss_D: 1.1936 Loss_G 0.4950 D(x): 0.3779 D(G(z)): 0.0447/0.6399
[4/5] [750/1583] Loss_D: 0.6820 Loss_G 2.3641 D(x): 0.7134 D(G(z)): 0.2400/0.1201
[4/5] [800/1583] Loss_D: 0.7211 Loss_G 3.0129 D(x): 0.9204 D(G(z)): 0.4249/0.0648
[4/5] [850/1583] Loss_D: 0.9899 Loss_G 3.3069 D(x): 0.8724 D(G(z)): 0.5214/0.0492
[4/5] [900/1583] Loss_D: 0.5789 Loss_G 2.5141 D(x): 0.7435 D(G(z)): 0.2110/0.1052
[4/5] [950/1583] Loss_D: 0.7162 Loss_G 1.3583 D(x): 0.5589 D(G(z)): 0.0576/0.3125
[4/5] [1000/1583] Loss_D: 1.1378 Loss_G 3.7072 D(x): 0.8517 D(G(z)): 0.5624/0.0364
[4/5] [1050/1583] Loss_D: 0.5823 Loss_G 2.5660 D(x): 0.7596 D(G(z)): 0.2257/0.0966
[4/5] [1100/1583] Loss_D: 0.7205 Loss_G 1.8147 D(x): 0.6805 D(G(z)): 0.2338/0.1991
[4/5] [1150/1583] Loss_D: 0.6265 Loss_G 2.7900 D(x): 0.7949 D(G(z)): 0.2872/0.0816
[4/5] [1200/1583] Loss_D: 1.1111 Loss_G 4.4571 D(x): 0.9287 D(G(z)): 0.5991/0.0167
[4/5] [1250/1583] Loss_D: 1.0609 Loss_G 4.3863 D(x): 0.8724 D(G(z)): 0.5500/0.0174
[4/5] [1300/1583] Loss_D: 0.6351 Loss_G 1.9326 D(x): 0.7810 D(G(z)): 0.2821/0.1783
[4/5] [1350/1583] Loss_D: 0.5135 Loss_G 2.3507 D(x): 0.7324 D(G(z)): 0.1416/0.1288
[4/5] [1400/1583] Loss_D: 0.6132 Loss_G 5.0354 D(x): 0.9302 D(G(z)): 0.3841/0.0102
[4/5] [1450/1583] Loss_D: 0.5440 Loss_G 2.3178 D(x): 0.7050 D(G(z)): 0.1354/0.1257
[4/5] [1500/1583] Loss_D: 0.5710 Loss_G 2.4214 D(x): 0.8401 D(G(z)): 0.2911/0.1163
[4/5] [1550/1583] Loss_D: 2.0148 Loss_G 4.4395 D(x): 0.9461 D(G(z)): 0.7895/0.0236

1
2
3

with torch.no_grad():
    fake = netG(fixed_noise).detach().cpu()
# fake

real_batch = next(iter(dataloader))

# Plot the real images
plt.figure(figsize=(30,30))
plt.subplot(1,2,1)
plt.axis=("off")
plt.title("Real Images")
plt.imshow(np.transpose(vutils.make_grid(real_batch[0].to(device)[:64], padding=5, normalize=True).cpu(),(1,2,0)))

# Plot the fake images from the last epoch
plt.subplot(1,2,2)
plt.axis=("off")
plt.title("Fake Images")
plt.imshow(np.transpose(vutils.make_grid(fake, padding=2, normalize=True), (1,2,0)))
plt.show()

png

5.CNN-Image-Classification

2020-02-29

第五课 CNN图像分类

褚则伟 zeweichu@gmail.com

参考资料

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
print("PyTorch Version: ",torch.__version__)

PyTorch Version:  1.0.0

首先我们定义一个基于ConvNet的简单神经网络

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1) # 28 * 28 -> (28+1-5) 24 * 24
        self.conv2 = nn.Conv2d(20, 50, 5, 1) # 20 * 20
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        # x: 1 * 28 * 28
        x = F.relu(self.conv1(x)) # 20 * 24 * 24
        x = F.max_pool2d(x,2,2) # 12 * 12
        x = F.relu(self.conv2(x)) # 8 * 8
        x = F.max_pool2d(x,2,2) # 4 *4
        x = x.view(-1, 4*4*50) # reshape (5 * 2 * 10), view(5, 20) -> (5 * 20)
        x = F.relu(self.fc1(x))
        x= self.fc2(x)
        # return x
        return F.log_softmax(x, dim=1) # log probability

mnist_data = datasets.MNIST("./mnist_data", train=True, download=True,
                           transform=transforms.Compose([
                               transforms.ToTensor(),
                           ]))
mnist_data

<torchvision.datasets.mnist.MNIST at 0x7fa362b9c7b8>

1	data = [d[0].data.cpu().numpy() for d in mnist_data]

1	np.mean(data)

0.13066062

1	np.std(data)

0.30810776

1	mnist_data[223][0].shape

torch.Size([1, 28, 28])

def train(model, device, train_loader, optimizer, epoch):
    model.train()
    for idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)

        pred = model(data) # batch_size * 10
        loss = F.nll_loss(pred, target)

        # SGD
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if idx % 100 == 0:
            print("Train Epoch: {}, iteration: {}, Loss: {}".format(
                epoch, idx, loss.item()))

def test(model, device, test_loader):
    model.eval()
    total_loss = 0.
    correct = 0.
    with torch.no_grad():
        for idx, (data, target) in enumerate(test_loader):
            data, target = data.to(device), target.to(device)

            output = model(data) # batch_size * 10
            total_loss += F.nll_loss(output, target, reduction="sum").item()
            pred = output.argmax(dim=1) # batch_size * 1
            correct += pred.eq(target.view_as(pred)).sum().item()


    total_loss /= len(test_loader.dataset)
    acc = correct/len(test_loader.dataset) * 100.
    print("Test loss: {}, Accuracy: {}".format(total_loss, acc))

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
batch_size = 32
train_dataloader = torch.utils.data.DataLoader(
    datasets.MNIST("./mnist_data", train=True, download=True,
           transform=transforms.Compose([
               transforms.ToTensor(),
               transforms.Normalize((0.1307,), (0.3081,))
           ])),
    batch_size=batch_size, shuffle=True,
    num_workers=1, pin_memory=True
)
test_dataloader = torch.utils.data.DataLoader(
    datasets.MNIST("./mnist_data", train=False, download=True,
           transform=transforms.Compose([
               transforms.ToTensor(),
               transforms.Normalize((0.1307,), (0.3081,))
           ])),
    batch_size=batch_size, shuffle=True,
    num_workers=1, pin_memory=True
)

lr = 0.01
momentum  = 0.5
model = Net().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum)

num_epochs = 2
for epoch in range(num_epochs):
    train(model, device, train_dataloader, optimizer, epoch)
    test(model, device, test_dataloader)

torch.save(model.state_dict(), "mnist_cnn.pt")

Train Epoch: 0, iteration: 0, Loss: 2.283817768096924
Train Epoch: 0, iteration: 100, Loss: 0.6110288500785828
Train Epoch: 0, iteration: 200, Loss: 0.18155980110168457
Train Epoch: 0, iteration: 300, Loss: 0.31043028831481934
Train Epoch: 0, iteration: 400, Loss: 0.518582284450531
Train Epoch: 0, iteration: 500, Loss: 0.1202855259180069
Train Epoch: 0, iteration: 600, Loss: 0.0989612340927124
Train Epoch: 0, iteration: 700, Loss: 0.09637182205915451
Train Epoch: 0, iteration: 800, Loss: 0.13470694422721863
Train Epoch: 0, iteration: 900, Loss: 0.06548292934894562
Train Epoch: 0, iteration: 1000, Loss: 0.03107370436191559
Train Epoch: 0, iteration: 1100, Loss: 0.03948028385639191
Train Epoch: 0, iteration: 1200, Loss: 0.09810394793748856
Train Epoch: 0, iteration: 1300, Loss: 0.15199752151966095
Train Epoch: 0, iteration: 1400, Loss: 0.016710489988327026
Train Epoch: 0, iteration: 1500, Loss: 0.005827277898788452
Train Epoch: 0, iteration: 1600, Loss: 0.0754864513874054
Train Epoch: 0, iteration: 1700, Loss: 0.012112855911254883
Train Epoch: 0, iteration: 1800, Loss: 0.03425520658493042
Test loss: 0.07333157858848571, Accuracy: 97.71
Train Epoch: 1, iteration: 0, Loss: 0.07740284502506256
Train Epoch: 1, iteration: 100, Loss: 0.018157958984375
Train Epoch: 1, iteration: 200, Loss: 0.006041824817657471
Train Epoch: 1, iteration: 300, Loss: 0.1392734944820404
Train Epoch: 1, iteration: 400, Loss: 0.022600188851356506
Train Epoch: 1, iteration: 500, Loss: 0.020594105124473572
Train Epoch: 1, iteration: 600, Loss: 0.031451016664505005
Train Epoch: 1, iteration: 700, Loss: 0.09078143537044525
Train Epoch: 1, iteration: 800, Loss: 0.013186424970626831
Train Epoch: 1, iteration: 900, Loss: 0.04006651043891907
Train Epoch: 1, iteration: 1000, Loss: 0.014285147190093994
Train Epoch: 1, iteration: 1100, Loss: 0.22637280821800232
Train Epoch: 1, iteration: 1200, Loss: 0.02185329794883728
Train Epoch: 1, iteration: 1300, Loss: 0.13519427180290222
Train Epoch: 1, iteration: 1400, Loss: 0.021606311202049255
Train Epoch: 1, iteration: 1500, Loss: 0.016718149185180664
Train Epoch: 1, iteration: 1600, Loss: 0.07150381058454514
Train Epoch: 1, iteration: 1700, Loss: 0.041178762912750244
Train Epoch: 1, iteration: 1800, Loss: 0.004264324903488159
Test loss: 0.040256525754928586, Accuracy: 98.61

NLL loss的定义

$\ell(x, y) = L = {l_1,\dots,l_N}^\top, \quad
l_n = - w_{y_n} x_{n,y_n}, \quad
w_{c} = \text{weight}[c] \cdot \mathbb{1}{c \not= \text{ignore_index}}$

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
batch_size = 32
train_dataloader = torch.utils.data.DataLoader(
    datasets.FashionMNIST("./fashion_mnist_data", train=True, download=True,
           transform=transforms.Compose([
               transforms.ToTensor(),
               transforms.Normalize((0.1307,), (0.3081,))
           ])),
    batch_size=batch_size, shuffle=True,
    num_workers=1, pin_memory=True
)
test_dataloader = torch.utils.data.DataLoader(
    datasets.FashionMNIST("./fashion_mnist_data", train=False, download=True,
           transform=transforms.Compose([
               transforms.ToTensor(),
               transforms.Normalize((0.1307,), (0.3081,))
           ])),
    batch_size=batch_size, shuffle=True,
    num_workers=1, pin_memory=True
)

lr = 0.01
momentum  = 0.5
model = Net().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=lr, momentum=momentum)

num_epochs = 2
for epoch in range(num_epochs):
    train(model, device, train_dataloader, optimizer, epoch)
    test(model, device, test_dataloader)

torch.save(model.state_dict(), "fashion_mnist_cnn.pt")

Train Epoch: 0, iteration: 0, Loss: 2.2915596961975098
Train Epoch: 0, iteration: 100, Loss: 1.0237065553665161
Train Epoch: 0, iteration: 200, Loss: 0.840910017490387
Train Epoch: 0, iteration: 300, Loss: 0.7526986598968506
Train Epoch: 0, iteration: 400, Loss: 0.9580956697463989
Train Epoch: 0, iteration: 500, Loss: 0.6261149644851685
Train Epoch: 0, iteration: 600, Loss: 0.4255485534667969
Train Epoch: 0, iteration: 700, Loss: 0.4818880558013916
Train Epoch: 0, iteration: 800, Loss: 0.731956958770752
Train Epoch: 0, iteration: 900, Loss: 0.45393142104148865
Train Epoch: 0, iteration: 1000, Loss: 0.7139236927032471
Train Epoch: 0, iteration: 1100, Loss: 0.4227047562599182
Train Epoch: 0, iteration: 1200, Loss: 0.23375816643238068
Train Epoch: 0, iteration: 1300, Loss: 0.4680781960487366
Train Epoch: 0, iteration: 1400, Loss: 0.352077841758728
Train Epoch: 0, iteration: 1500, Loss: 0.36358141899108887
Train Epoch: 0, iteration: 1600, Loss: 0.46214842796325684
Train Epoch: 0, iteration: 1700, Loss: 0.4750059247016907
Train Epoch: 0, iteration: 1800, Loss: 0.4483456015586853
Test loss: 0.45549455399513245, Accuracy: 83.48
Train Epoch: 1, iteration: 0, Loss: 0.46502870321273804
Train Epoch: 1, iteration: 100, Loss: 0.4504859745502472
Train Epoch: 1, iteration: 200, Loss: 0.5228638648986816
Train Epoch: 1, iteration: 300, Loss: 0.507514476776123
Train Epoch: 1, iteration: 400, Loss: 0.33425623178482056
Train Epoch: 1, iteration: 500, Loss: 0.15890713036060333
Train Epoch: 1, iteration: 600, Loss: 0.4329398274421692
Train Epoch: 1, iteration: 700, Loss: 0.47604358196258545
Train Epoch: 1, iteration: 800, Loss: 0.40596315264701843
Train Epoch: 1, iteration: 900, Loss: 0.31725335121154785
Train Epoch: 1, iteration: 1000, Loss: 0.5835919380187988
Train Epoch: 1, iteration: 1100, Loss: 0.3334502577781677
Train Epoch: 1, iteration: 1200, Loss: 0.3043973743915558
Train Epoch: 1, iteration: 1300, Loss: 0.3891294002532959
Train Epoch: 1, iteration: 1400, Loss: 0.20209042727947235
Train Epoch: 1, iteration: 1500, Loss: 0.26769235730171204
Train Epoch: 1, iteration: 1600, Loss: 0.366751104593277
Train Epoch: 1, iteration: 1700, Loss: 0.16336065530776978
Train Epoch: 1, iteration: 1800, Loss: 0.48901161551475525
Test loss: 0.37672775785923, Accuracy: 86.15

CNN模型的迁移学习

很多时候当我们需要训练一个新的图像分类任务，我们不会完全从一个随机的模型开始训练，而是利用_预训练_的模型来加速训练的过程。我们经常使用在ImageNet上的预训练模型。
这是一种transfer learning的方法。我们常用以下两种方法做迁移学习。
- fine tuning: 从一个预训练模型开始，我们改变一些模型的架构，然后继续训练整个模型的参数。
- feature extraction: 我们不再改变预训练模型的参数，而是只更新我们改变过的部分模型参数。我们之所以叫它feature extraction是因为我们把预训练的CNN模型当做一个特征提取模型，利用提取出来的特征做来完成我们的训练任务。

以下是构建和训练迁移学习模型的基本步骤：

初始化预训练模型
把最后一层的输出层改变成我们想要分的类别总数
定义一个optimizer来更新参数
模型训练

import numpy as np
import torchvision
from torchvision import datasets, transforms, models

import matplotlib.pyplot as plt
import time
import os
import copy
print("Torchvision Version: ",torchvision.__version__)

Torchvision Version:  0.2.0

数据

我们会使用hymenoptera_data数据集，下载.

这个数据集包括两类图片, bees 和 ants, 这些数据都被处理成了可以使用ImageFolder <https://pytorch.org/docs/stable/torchvision/datasets.html#torchvision.datasets.ImageFolder>来读取的格式。我们只需要把data_dir设置成数据的根目录，然后把model_name设置成我们想要使用的与训练模型：
::
[resnet, alexnet, vgg, squeezenet, densenet, inception]

其他的参数有：

num_classes表示数据集分类的类别数
batch_size
num_epochs
feature_extract表示我们训练的时候使用fine tuning还是feature extraction方法。如果feature_extract = False，整个模型都会被同时更新。如果feature_extract = True，只有模型的最后一层被更新。

# Top level data directory. Here we assume the format of the directory conforms
#   to the ImageFolder structure
data_dir = "./hymenoptera_data"
# Models to choose from [resnet, alexnet, vgg, squeezenet, densenet, inception]
model_name = "resnet"
# Number of classes in the dataset
num_classes = 2
# Batch size for training (change depending on how much memory you have)
batch_size = 32
# Number of epochs to train for
num_epochs = 15
# Flag for feature extracting. When False, we finetune the whole model,
#   when True we only update the reshaped layer params
feature_extract = True

input_size = 224

读入数据

现在我们知道了模型输入的size，我们就可以把数据预处理成相应的格式。

all_imgs = datasets.ImageFolder(os.path.join(data_dir, "train"), transforms.Compose([
        transforms.RandomResizedCrop(input_size),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
    ]))
loader = torch.utils.data.DataLoader(all_imgs, batch_size=batch_size, shuffle=True, num_workers=4)

data_transforms = {
    "train": transforms.Compose([
        transforms.RandomResizedCrop(input_size),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    "val": transforms.Compose([
        transforms.Resize(input_size),
        transforms.CenterCrop(input_size),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ])
}

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ["train", "val"]}

dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x],
        batch_size=batch_size, shuffle=True, num_workers=4) for x in ["train", "val"]}

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

1	img = next(iter(dataloaders_dict["val"]))[0]

img.shape

torch.Size([32, 3, 224, 224])

unloader = transforms.ToPILImage()  # reconvert into PIL image

plt.ion()

def imshow(tensor, title=None):
    image = tensor.cpu().clone()  # we clone the tensor to not do changes on it
    image = image.squeeze(0)      # remove the fake batch dimension
    image = unloader(image)
    plt.imshow(image)
    if title is not None:
        plt.title(title)
    plt.pause(0.001) # pause a bit so that plots are updated


plt.figure()
imshow(img[11], title='Image')

png

def set_parameter_requires_grad(model, feature_extract):
    if feature_extract:
        for param in model.parameters():
            param.requires_grad = False

def initialize_model(model_name, num_classes, feature_extract, use_pretrained=True):
    if model_name == "resnet":
        model_ft = models.resnet18(pretrained=use_pretrained)
        set_parameter_requires_grad(model_ft, feature_extract)
        num_ftrs = model_ft.fc.in_features
        model_ft.fc = nn.Linear(num_ftrs, num_classes)
        input_size = 224
    else:
        print("model not implemented")
        return None, None

    return model_ft, input_size

model_ft, input_size = initialize_model(model_name,
                    num_classes, feature_extract, use_pretrained=True)
print(model_ft)

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer2): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (layer4): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AvgPool2d(kernel_size=7, stride=1, padding=0)
  (fc): Linear(in_features=512, out_features=2, bias=True)
)

1	model_ft.layer1[0].conv1.weight.requires_grad

False

1	model_ft.fc.weight.requires_grad

True

def train_model(model, dataloaders, loss_fn, optimizer, num_epochs=5):
    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.
    val_acc_history = []
    for epoch in range(num_epochs):
        for phase in ["train", "val"]:
            running_loss = 0.
            running_corrects = 0.
            if phase == "train":
                model.train()
            else:
                model.eval()

            for inputs, labels in dataloaders[phase]:
                inputs, labels = inputs.to(device), labels.to(device)

                with torch.autograd.set_grad_enabled(phase=="train"):
                    outputs = model(inputs) # bsize * 2
                    loss = loss_fn(outputs, labels)

                preds = outputs.argmax(dim=1)
                if phase == "train":
                    optimizer.zero_grad()
                    loss.backward()
                    optimizer.step()
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds.view(-1) == labels.view(-1)).item()

            epoch_loss = running_loss / len(dataloaders[phase].dataset)
            epoch_acc = running_corrects / len(dataloaders[phase].dataset)

            print("Phase {} loss: {}, acc: {}".format(phase, epoch_loss, epoch_acc))

            if phase == "val" and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
            if phase == "val":
                val_acc_history.append(epoch_acc)
    model.load_state_dict(best_model_wts)    
    return model, val_acc_history

模型训练

1	model_ft = model_ft.to(device)

1
2
3

optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad,
                                   model_ft.parameters()), lr=0.001, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()

1	_, ohist = train_model(model_ft, dataloaders_dict, loss_fn, optimizer, num_epochs=num_epochs)

Phase train loss: 0.2441009590860273, acc: 0.9139344262295082
Phase val loss: 0.2058036023495244, acc: 0.9477124183006536
Phase train loss: 0.2242034280397853, acc: 0.9221311475409836
Phase val loss: 0.19933121480972937, acc: 0.9477124183006536
Phase train loss: 0.2179500304284643, acc: 0.930327868852459
Phase val loss: 0.19292019200480842, acc: 0.9477124183006536
Phase train loss: 0.2038783010889272, acc: 0.9221311475409836
Phase val loss: 0.2022019473750607, acc: 0.9281045751633987
Phase train loss: 0.20605210031642288, acc: 0.9180327868852459
Phase val loss: 0.18852663916700027, acc: 0.9477124183006536
Phase train loss: 0.1799576844348282, acc: 0.9426229508196722
Phase val loss: 0.18889451397010704, acc: 0.9477124183006536
Phase train loss: 0.16676783659419075, acc: 0.9426229508196722
Phase val loss: 0.1854035053280444, acc: 0.9477124183006536
Phase train loss: 0.20258395642530722, acc: 0.930327868852459
Phase val loss: 0.1881853450162738, acc: 0.934640522875817
Phase train loss: 0.17906492948532104, acc: 0.9180327868852459
Phase val loss: 0.17941297795258315, acc: 0.954248366013072
Phase train loss: 0.15364321333463074, acc: 0.9631147540983607
Phase val loss: 0.18958801722604465, acc: 0.9281045751633987
Phase train loss: 0.19896865452899307, acc: 0.9139344262295082
Phase val loss: 0.1826314626176373, acc: 0.954248366013072
Phase train loss: 0.16911793878821077, acc: 0.9344262295081968
Phase val loss: 0.18108942452209448, acc: 0.9477124183006536
Phase train loss: 0.16306845306373033, acc: 0.9467213114754098
Phase val loss: 0.1891336505806524, acc: 0.9281045751633987
Phase train loss: 0.1875694076545903, acc: 0.9385245901639344
Phase val loss: 0.1793875343659345, acc: 0.9477124183006536
Phase train loss: 0.20147151096922453, acc: 0.9139344262295082
Phase val loss: 0.18119409422274507, acc: 0.9411764705882353

model_scratch, _ = initialize_model(model_name,
                    num_classes, feature_extract=False, use_pretrained=False)
model_scratch = model_scratch.to(device)
optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad,
                                   model_scratch.parameters()), lr=0.001, momentum=0.9)
loss_fn = nn.CrossEntropyLoss()
_, scratch_hist = train_model(model_scratch, dataloaders_dict, loss_fn, optimizer, num_epochs=num_epochs)

Phase train loss: 0.73858080437926, acc: 0.47950819672131145
Phase val loss: 0.704963364632301, acc: 0.46405228758169936
Phase train loss: 0.668987612255284, acc: 0.5614754098360656
Phase val loss: 0.6597700851415497, acc: 0.6339869281045751
Phase train loss: 0.6411691278707786, acc: 0.6557377049180327
Phase val loss: 0.6726375681902069, acc: 0.5751633986928104
Phase train loss: 0.6194371883986426, acc: 0.6352459016393442
Phase val loss: 0.6313814318257999, acc: 0.6274509803921569
Phase train loss: 0.6170555851498588, acc: 0.6475409836065574
Phase val loss: 0.6528662945709977, acc: 0.6274509803921569
Phase train loss: 0.6110719637792619, acc: 0.6762295081967213
Phase val loss: 0.626404657472972, acc: 0.6405228758169934
Phase train loss: 0.5864127718034338, acc: 0.639344262295082
Phase val loss: 0.6282203664966658, acc: 0.6405228758169934
Phase train loss: 0.5998562847981688, acc: 0.6680327868852459
Phase val loss: 0.6236733716297773, acc: 0.6274509803921569
Phase train loss: 0.5662176755608105, acc: 0.6885245901639344
Phase val loss: 0.5790788285872516, acc: 0.6862745098039216
Phase train loss: 0.5466401464626437, acc: 0.7131147540983607
Phase val loss: 0.5834652006236556, acc: 0.7124183006535948
Phase train loss: 0.5393341779708862, acc: 0.7295081967213115
Phase val loss: 0.5651591182534211, acc: 0.6797385620915033
Phase train loss: 0.5473490689621597, acc: 0.7172131147540983
Phase val loss: 0.5568503246587866, acc: 0.673202614379085
Phase train loss: 0.5429437048122531, acc: 0.7090163934426229
Phase val loss: 0.6801646998505188, acc: 0.6339869281045751
Phase train loss: 0.512938850238675, acc: 0.7254098360655737
Phase val loss: 0.6064363223275328, acc: 0.6862745098039216
Phase train loss: 0.5331279508403091, acc: 0.6885245901639344
Phase val loss: 0.5726334435098311, acc: 0.6928104575163399

我们来plot模型训练时候loss的变化


# Plot the training curves of validation accuracy vs. number
#  of training epochs for the transfer learning method and
#  the model trained from scratch

plt.title("Validation Accuracy vs. Number of Training Epochs")
plt.xlabel("Training Epochs")
plt.ylabel("Validation Accuracy")
plt.plot(range(1,num_epochs+1),ohist,label="Pretrained")
plt.plot(range(1,num_epochs+1),scratch_hist,label="Scratch")
plt.ylim((0,1.))
plt.xticks(np.arange(1, num_epochs+1, 1.0))
plt.legend()
plt.show()

png

课后学习

1.two_layer_neural_net

2020-02-29

第一课

参考资料 reference

什么是PyTorch?

PyTorch是一个基于Python的科学计算库，它有以下特点:

类似于NumPy，但是它可以使用GPU
可以用它定义深度学习模型，可以灵活地进行深度学习模型的训练和使用

Tensors

Tensor类似与NumPy的ndarray，唯一的区别是Tensor可以在GPU上加速运算。

1 2	from __future__ import print_function import torch

构造一个未初始化的5x3矩阵:

1 2	x = torch.empty(5, 3) print(x)

tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00],
        [0.0000e+00, 0.0000e+00, 1.1704e-41],
        [0.0000e+00, 2.2369e+08, 0.0000e+00],
        [0.0000e+00, 0.0000e+00, 0.0000e+00],
        [       nan,        nan, 5.8024e-08]])

构建一个随机初始化的矩阵:

1 2	x = torch.rand(5, 3) print(x)

tensor([[0.8864, 0.0875, 0.5509],
        [0.8290, 0.4770, 0.5214],
        [0.4291, 0.8248, 0.7763],
        [0.8400, 0.4375, 0.8075],
        [0.4034, 0.8427, 0.6798]])

构建一个全部为0，类型为long的矩阵:

1 2	x = torch.zeros(5, 3, dtype=torch.long) print(x)

tensor([[0, 0, 0],
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0],
        [0, 0, 0]])

从数据直接直接构建tensor:

1 2	x = torch.tensor([5.5, 3]) print(x)

tensor([5.5000, 3.0000])

也可以从一个已有的tensor构建一个tensor。这些方法会重用原来tensor的特征，例如，数据类型，除非提供新的数据。

x = x.new_ones(5, 3, dtype=torch.double)      # new_* methods take in sizes
print(x)

x = torch.randn_like(x, dtype=torch.float)    # override dtype!
print(x)                                      # result has the same size

tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]], dtype=torch.float64)
tensor([[ 0.2173, -1.3224,  0.5543],
        [ 0.1450,  0.2199, -0.0678],
        [ 0.7979, -0.4419, -1.2025],
        [ 0.8140,  0.2363,  0.2310],
        [-1.7360,  1.9243,  1.5151]])

得到tensor的形状:

1	print(x.size())

torch.Size([5, 3])

注意

``torch.Size`` 返回的是一个tuple

Operations

有很多种tensor运算。我们先介绍加法运算。

1 2	y = torch.rand(5, 3) print(x + y)

tensor([[ 0.6617, -0.9648,  1.4188],
        [ 0.4702,  0.6419,  0.6207],
        [ 1.5369,  0.2954, -0.7771],
        [ 1.4111,  0.9671,  1.0595],
        [-1.3126,  2.2809,  1.8937]])

另一种着加法的写法

1	print(torch.add(x, y))

tensor([[ 0.6617, -0.9648,  1.4188],
        [ 0.4702,  0.6419,  0.6207],
        [ 1.5369,  0.2954, -0.7771],
        [ 1.4111,  0.9671,  1.0595],
        [-1.3126,  2.2809,  1.8937]])

加法：把输出作为一个变量

1
2
3

result = torch.empty(5, 3)
torch.add(x, y, out=result)
print(result)

tensor([[ 0.6617, -0.9648,  1.4188],
        [ 0.4702,  0.6419,  0.6207],
        [ 1.5369,  0.2954, -0.7771],
        [ 1.4111,  0.9671,  1.0595],
        [-1.3126,  2.2809,  1.8937]])

in-place加法

1
2
3

# adds x to y
y.add_(x)
print(y)

tensor([[ 0.6617, -0.9648,  1.4188],
        [ 0.4702,  0.6419,  0.6207],
        [ 1.5369,  0.2954, -0.7771],
        [ 1.4111,  0.9671,  1.0595],
        [-1.3126,  2.2809,  1.8937]])

注意

任何in-place的运算都会以``_``结尾。举例来说：``x.copy_(y)``, ``x.t_()``, 会改变 ``x``。

各种类似NumPy的indexing都可以在PyTorch tensor上面使用。

1	print(x[:, 1])

tensor([-1.3224,  0.2199, -0.4419,  0.2363,  1.9243])

Resizing: 如果你希望resize/reshape一个tensor，可以使用torch.view：

x = torch.randn(4, 4)
y = x.view(16)
z = x.view(-1, 8)  # the size -1 is inferred from other dimensions
print(x.size(), y.size(), z.size())

torch.Size([4, 4]) torch.Size([16]) torch.Size([2, 8])

如果你有一个只有一个元素的tensor，使用.item()方法可以把里面的value变成Python数值。

1
2
3

x = torch.randn(1)
print(x)
print(x.item())

tensor([0.5669])
0.5668683052062988

更多阅读

各种Tensor operations, 包括transposing, indexing, slicing,
mathematical operations, linear algebra, random numbers在
<https://pytorch.org/docs/torch>.

Numpy和Tensor之间的转化

在Torch Tensor和NumPy array之间相互转化非常容易。

Torch Tensor和NumPy array会共享内存，所以改变其中一项也会改变另一项。

把Torch Tensor转变成NumPy Array

1 2	a = torch.ones(5) print(a)

tensor([1., 1., 1., 1., 1.])

1 2	b = a.numpy() print(b)

[1. 1. 1. 1. 1.]

改变numpy array里面的值。

1
2
3

a.add_(1)
print(a)
print(b)

tensor([2., 2., 2., 2., 2.])
[2. 2. 2. 2. 2.]

把NumPy ndarray转成Torch Tensor

import numpy as np
a = np.ones(5)
b = torch.from_numpy(a)
np.add(a, 1, out=a)
print(a)
print(b)

[2. 2. 2. 2. 2.]
tensor([2., 2., 2., 2., 2.], dtype=torch.float64)

所有CPU上的Tensor都支持转成numpy或者从numpy转成Tensor。

CUDA Tensors

使用.to方法，Tensor可以被移动到别的device上。

# let us run this cell only if CUDA is available
# We will use ``torch.device`` objects to move tensors in and out of GPU
if torch.cuda.is_available():
    device = torch.device("cuda")          # a CUDA device object
    y = torch.ones_like(x, device=device)  # directly create a tensor on GPU
    x = x.to(device)                       # or just use strings ``.to("cuda")``
    z = x + y
    print(z)
    print(z.to("cpu", torch.double))       # ``.to`` can also change dtype together!

热身: 用numpy实现两层神经网络

一个全连接ReLU神经网络，一个隐藏层，没有bias。用来从x预测y，使用L2 Loss。

这一实现完全使用numpy来计算前向神经网络，loss，和反向传播。

numpy ndarray是一个普通的n维array。它不知道任何关于深度学习或者梯度(gradient)的知识，也不知道计算图(computation graph)，只是一种用来计算数学运算的数据结构。

import numpy as np

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = np.random.randn(N, D_in)
y = np.random.randn(N, D_out)

# Randomly initialize weights
w1 = np.random.randn(D_in, H)
w2 = np.random.randn(H, D_out)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.dot(w1)
    h_relu = np.maximum(h, 0)
    y_pred = h_relu.dot(w2)

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss

    # loss = (y_pred - y) ** 2
    grad_y_pred = 2.0 * (y_pred - y)
    #
    grad_w2 = h_relu.T.dot(grad_y_pred)
    grad_h_relu = grad_y_pred.dot(w2.T)
    grad_h = grad_h_relu.copy()
    grad_h[h < 0] = 0
    grad_w1 = x.T.dot(grad_h)

    # Update weights
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

0 35889496.41026865
1 28319860.01378521
2 22126579.361950204
3 15942003.17226502
4 10582823.172692467
5 6712365.992110595
6 4278837.734943641
7 2840714.133008884
8 2000039.205437642
9 1489169.222664059
10 1160366.8418461317
11 934585.7940074136
12 770239.7520808787
13 645310.3968864691
14 547178.2130908443
15 468305.96276602603
16 403720.3323130417
17 350149.3157995759
18 305257.8820296975
19 267369.8879260684
20 235109.93680038114
21 207505.89253678697
22 183800.59602622912
23 163307.92930747362
24 145544.49787693794
25 130055.24947615515
26 116503.63761546681
27 104610.98914657751
28 94143.48296673188
29 84901.07887630322
30 76740.20444201502
31 69484.37249565196
32 63017.32854658759
33 57241.53192189535
34 52072.6557028254
35 47437.10801676742
36 43270.66165969739
37 39520.47017020389
38 36137.87167372235
39 33082.18308102614
40 30317.817057848697
41 27812.480443743487
42 25542.70294062024
43 23482.66570167685
44 21610.55239986089
45 19906.160873831992
46 18351.51791836552
47 16932.27679740901
48 15634.311658318817
49 14447.479800214693
50 13360.284467202091
51 12364.55703521107
52 11450.507526015177
53 10610.716759762126
54 9838.763456506194
55 9127.861849646964
56 8473.405368145268
57 7870.267914674974
58 7313.881129291616
59 6800.372174677839
60 6325.8330150692855
61 5887.136534392823
62 5481.921842552875
63 5106.981169080381
64 4759.885185719859
65 4438.06608714836
66 4139.591989541515
67 3862.801801341933
68 3605.722147246028
69 3367.0466574316465
70 3145.2469158861986
71 2939.0792653683548
72 2747.4439735106916
73 2568.926300863709
74 2402.7727293645053
75 2248.061221973625
76 2103.9061542778472
77 1969.5776758477564
78 1844.278108461921
79 1727.4206281578204
80 1618.438218620885
81 1516.629044425202
82 1421.6386822769778
83 1332.8911655138672
84 1249.8827229524381
85 1172.3145094934905
86 1099.7771973726194
87 1031.9358943423872
88 968.5315238801331
89 909.1833300753982
90 853.6103523323618
91 801.6057773802581
92 752.8972257884798
93 707.2716083720514
94 664.5922195801298
95 624.585130687555
96 587.1062695014971
97 551.9631328777837
98 519.0004053966405
99 488.09296557554535
100 459.09994393346363
101 431.8708781694225
102 406.33133782098787
103 382.35649672519094
104 359.84580378688275
105 338.70144913768877
106 318.84518288061736
107 300.1785850987517
108 282.64617492915346
109 266.1687762171392
110 250.67644576515903
111 236.11921822004882
112 222.4281934174508
113 209.55112223597783
114 197.44940642456191
115 186.06234551368843
116 175.34381840002771
117 165.26565179843442
118 155.78056310618527
119 146.850513943774
120 138.44727840131497
121 130.53969525793087
122 123.09136097849083
123 116.08012321986509
124 109.48025541067466
125 103.2596385459983
126 97.39972134228576
127 91.88505267835866
128 86.68592773077614
129 81.78849775294941
130 77.17633325116682
131 72.8255347502833
132 68.72580481324971
133 64.86409871447643
134 61.2214114158698
135 57.78627868853266
136 54.549203258015424
137 51.49631802973718
138 48.616519517573636
139 45.90288571497679
140 43.34285956860195
141 40.92689419975899
142 38.64784156896851
143 36.49942566295091
144 34.47060829150904
145 32.55634851157603
146 30.751409245270633
147 29.04709820655909
148 27.43852093668076
149 25.921388411314368
150 24.489148370444664
151 23.136659002969125
152 21.860173471455816
153 20.655812309970656
154 19.51776263502822
155 18.44338176312585
156 17.430044604727087
157 16.47242130321127
158 15.568835000405398
159 14.716219937809395
160 13.911554249638831
161 13.153837764734106
162 12.43835150933005
163 11.762430963758575
164 11.123342669940381
165 10.51955406473213
166 9.949353731973357
167 9.410161158092585
168 8.900781237692307
169 8.419047503930305
170 7.963841961182751
171 7.533284443766788
172 7.126285614755901
173 6.7419926627131765
174 6.378370063846165
175 6.034550928076551
176 5.709438929865164
177 5.402012458870738
178 5.1112581148191785
179 4.836315288679497
180 4.576507698505559
181 4.330668987833015
182 4.098192919994734
183 3.878308283127276
184 3.670289507272357
185 3.473443405932639
186 3.287310001550204
187 3.1113456208500265
188 2.944787543919021
189 2.7873138898506102
190 2.6383954264383527
191 2.4973805234910067
192 2.3639071972119474
193 2.237667463271773
194 2.1182887816264833
195 2.0052490195791135
196 1.8983723966561192
197 1.7972572916391805
198 1.7014737913249238
199 1.6108294851492475
200 1.5251066392156072
201 1.4439797000391867
202 1.367156696490094
203 1.2944797363759633
204 1.2257763592421744
205 1.1606420258559709
206 1.0989863637045116
207 1.0406888635926097
208 0.985480656856923
209 0.9331948690786269
210 0.883708107992991
211 0.8369371951471725
212 0.7925830252882534
213 0.7505963077993925
214 0.7108639048331671
215 0.6732499797461134
216 0.6376217188582812
217 0.6038948816719618
218 0.5720028342100039
219 0.5417794097937626
220 0.5131446427202526
221 0.48604817187601246
222 0.4603925764718072
223 0.43608421638073835
224 0.4130783939606849
225 0.3913067029399362
226 0.37068242884203506
227 0.3511394228583754
228 0.3326419235697783
229 0.3151202969121928
230 0.2985201427168679
231 0.28280176099237087
232 0.26792719479470506
233 0.2538283345177868
234 0.24048493479252905
235 0.2278397120969069
236 0.21586361679459082
237 0.20451607817993503
238 0.193766188427152
239 0.1835939793360537
240 0.17395143814600034
241 0.16482273829369665
242 0.15618115286925285
243 0.14798768376692845
244 0.14022225707294186
245 0.13286675572070444
246 0.12590636126446822
247 0.11930498471500077
248 0.11305144681669504
249 0.10713339447378334
250 0.10152582283064925
251 0.09620851818029758
252 0.09117062929007841
253 0.08640227528150915
254 0.08187998430831803
255 0.07759528862448145
256 0.07353807496661162
257 0.06969661935521981
258 0.06605339247911975
259 0.06260173590955692
260 0.05933375737331758
261 0.056233423718695594
262 0.05329591100700592
263 0.05051351071970529
264 0.04787721981662252
265 0.0453796648621478
266 0.04301134709440588
267 0.040769094283493554
268 0.0386419065745876
269 0.036626397581498354
270 0.03471769463128084
271 0.03290833008693546
272 0.03119283555474027
273 0.029568770350473256
274 0.02802936482297661
275 0.026569462253079254
276 0.025185751177444198
277 0.02387540559002874
278 0.02263296632399935
279 0.021454766168038826
280 0.020338411270745704
281 0.01928209637734954
282 0.01827890124684954
283 0.017328178012093416
284 0.016427431131624558
285 0.015573733337988723
286 0.014764164824078314
287 0.013996810616742416
288 0.013270192784228377
289 0.012581339060228808
290 0.011927855449286724
291 0.01130885579850701
292 0.010721954587038451
293 0.010165289932731865
294 0.009637652275353552
295 0.009138171497490244
296 0.008664015927898068
297 0.008214956270036388
298 0.007789027990233297
299 0.00738526092049741
300 0.0070023116518919334
301 0.0066393077276630874
302 0.006295496247275727
303 0.005969185383562498
304 0.005659890415241421
305 0.0053670577521739175
306 0.005089192297943729
307 0.004825661143498267
308 0.0045757630820664865
309 0.004339075991432088
310 0.004114455095453374
311 0.003901527998374378
312 0.003699735383802623
313 0.003508615907063876
314 0.0033271132857261515
315 0.0031550406159224515
316 0.0029920362648068006
317 0.002837311781147677
318 0.002690614891073643
319 0.002551584154582522
320 0.002419787803521223
321 0.0022948269249008675
322 0.0021762725948545484
323 0.002063963234582214
324 0.0019573609105524746
325 0.0018562702760715648
326 0.0017604606858835246
327 0.0016696047989841608
328 0.0015834097092508315
329 0.0015017298703711534
330 0.001424349420552173
331 0.0013508712009882061
332 0.0012811738966555235
333 0.001215120977868637
334 0.0011524767424944495
335 0.0010930402668986058
336 0.0010366787893071614
337 0.0009832899731449837
338 0.000932639857726871
339 0.0008845714854927703
340 0.0008390201257023468
341 0.0007958028124191009
342 0.0007548028166217269
343 0.0007159268548135305
344 0.0006790821656325096
345 0.0006441073800533147
346 0.0006109654113912357
347 0.0005795355926475646
348 0.0005497227132968953
349 0.0005214274073810845
350 0.000494598323300069
351 0.00046916693850541105
352 0.0004450259439185543
353 0.0004221310873514728
354 0.0004004414994868116
355 0.00037986226123455495
356 0.00036032844005961715
357 0.0003418078385461384
358 0.0003242479910687428
359 0.000307580574600084
360 0.000291771372027413
361 0.00027678752922397234
362 0.0002625666007376476
363 0.0002490890956713469
364 0.00023629671429026766
365 0.00022416908984249217
366 0.00021266049160700826
367 0.00020173979435604266
368 0.00019139048985311868
369 0.00018156454712891162
370 0.00017224458704855636
371 0.0001634118654761384
372 0.00015503237540818944
373 0.00014707671783726012
374 0.00013953112289743838
375 0.00013238040053695685
376 0.00012558966491342063
377 0.00011914786179227344
378 0.00011304139590044389
379 0.00010724684438892698
380 0.00010175233099436165
381 9.653729588396157e-05
382 9.159291113932786e-05
383 8.689974396932523e-05
384 8.244687248072015e-05
385 7.822567911872094e-05
386 7.421843362658021e-05
387 7.041600947384089e-05
388 6.681167652694762e-05
389 6.339366727411276e-05
390 6.014697337326878e-05
391 5.706694467227549e-05
392 5.414831349568961e-05
393 5.137673801360314e-05
394 4.8746787304773246e-05
395 4.625341761388583e-05
396 4.388730680241007e-05
397 4.164267190789709e-05
398 3.9513033436439895e-05
399 3.7493247913474995e-05
400 3.557521246939875e-05
401 3.375652186549214e-05
402 3.203146827189639e-05
403 3.0393805627207457e-05
404 2.883975535048804e-05
405 2.7365856401607758e-05
406 2.5968899386329313e-05
407 2.4641548369272432e-05
408 2.338213157757801e-05
409 2.218843584869327e-05
410 2.1054686194234953e-05
411 1.997897135307684e-05
412 1.895917609351847e-05
413 1.799105790586622e-05
414 1.7072338606631315e-05
415 1.620126489878264e-05
416 1.537456630903519e-05
417 1.4589560446195426e-05
418 1.3844903455289862e-05
419 1.3138944536107086e-05
420 1.2468462389327143e-05
421 1.1832124847189769e-05
422 1.1228621028826248e-05
423 1.0656066913018655e-05
424 1.011273346058932e-05
425 9.596919075218238e-06
426 9.107809268437839e-06
427 8.643283243121413e-06
428 8.202496216368065e-06
429 7.78453476735051e-06
430 7.387714887329486e-06
431 7.011127896690068e-06
432 6.6539493887270735e-06
433 6.315198709781024e-06
434 5.9933376724304165e-06
435 5.687928494267994e-06
436 5.398469646033336e-06
437 5.123455552248832e-06
438 4.862439363315032e-06
439 4.614923178641405e-06
440 4.379928467499698e-06
441 4.156881750155099e-06
442 3.945435030675091e-06
443 3.7446593517849704e-06
444 3.5540106822035164e-06
445 3.3730943350520115e-06
446 3.2015523080038885e-06
447 3.0386018502004967e-06
448 2.8839774095235675e-06
449 2.7373106877564923e-06
450 2.5980805389395875e-06
451 2.466016122933826e-06
452 2.3406133927609447e-06
453 2.2216389608068128e-06
454 2.1087008081973983e-06
455 2.0014582936333935e-06
456 1.8997777867189913e-06
457 1.8031838999248728e-06
458 1.7115102131443443e-06
459 1.624558464328827e-06
460 1.5420602925846657e-06
461 1.4636964584374802e-06
462 1.3893347296004668e-06
463 1.3187801768754607e-06
464 1.2517744421070682e-06
465 1.1881791041890356e-06
466 1.1278774051305736e-06
467 1.070585095611918e-06
468 1.0162097470498576e-06
469 9.646641556210136e-07
470 9.157031601476153e-07
471 8.692101272246917e-07
472 8.251294357060302e-07
473 7.832638296613223e-07
474 7.435047459907646e-07
475 7.057740445142993e-07
476 6.699852845013653e-07
477 6.359867970893319e-07
478 6.037292747642512e-07
479 5.73137535567069e-07
480 5.440663178206311e-07
481 5.164693271072091e-07
482 4.902873072466486e-07
483 4.6543234211654636e-07
484 4.418303204926691e-07
485 4.194335470102758e-07
486 3.9818141233304274e-07
487 3.779968526851402e-07
488 3.5884846089674434e-07
489 3.4067883723659903e-07
490 3.234212375174028e-07
491 3.07031155707031e-07
492 2.9148452610441727e-07
493 2.767219156168417e-07
494 2.6270341200559135e-07
495 2.4940159652727215e-07
496 2.3677534677773108e-07
497 2.2478718706326765e-07
498 2.1341070029951215e-07
499 2.026117831426625e-07

PyTorch: Tensors

这次我们使用PyTorch tensors来创建前向神经网络，计算损失，以及反向传播。

一个PyTorch Tensor很像一个numpy的ndarray。但是它和numpy ndarray最大的区别是，PyTorch Tensor可以在CPU或者GPU上运算。如果想要在GPU上运算，就需要把Tensor换成cuda类型。

import torch


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(500):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

0 28695648.0
1 25650756.0
2 27025320.0
3 28834564.0
4 27564222.0
5 22166346.0
6 14644791.0
7 8384613.0
8 4518598.0
9 2533808.0
10 1567210.875
11 1086797.875
12 826069.8125
13 666741.5625
14 557421.125
15 475584.75
16 410677.84375
17 357391.90625
18 312749.96875
19 274865.53125
20 242451.6875
21 214546.75
22 190410.90625
23 169438.953125
24 151177.625
25 135196.59375
26 121190.5
27 108860.890625
28 97984.0546875
29 88356.1796875
30 79816.2734375
31 72226.7109375
32 65471.046875
33 59439.18359375
34 54043.625
35 49208.23828125
36 44866.76953125
37 40961.1484375
38 37443.1328125
39 34269.6640625
40 31402.91015625
41 28808.783203125
42 26457.36328125
43 24324.15625
44 22385.1796875
45 20620.73828125
46 19013.07421875
47 17546.958984375
48 16208.193359375
49 14984.5576171875
50 13864.78125
51 12838.96875
52 11898.251953125
53 11036.388671875
54 10244.4775390625
55 9515.7958984375
56 8845.623046875
57 8227.857421875
58 7657.828125
59 7131.6416015625
60 6645.55224609375
61 6196.15771484375
62 5780.22021484375
63 5394.966796875
64 5037.93017578125
65 4706.81689453125
66 4399.5654296875
67 4114.1865234375
68 3849.041748046875
69 3602.55810546875
70 3373.184326171875
71 3159.7119140625
72 2960.900390625
73 2775.63427734375
74 2602.867919921875
75 2441.733154296875
76 2291.387451171875
77 2151.02197265625
78 2019.9407958984375
79 1897.4022216796875
80 1782.8963623046875
81 1675.7479248046875
82 1575.53515625
83 1481.785888671875
84 1393.9783935546875
85 1311.72119140625
86 1234.640869140625
87 1162.418212890625
88 1094.647705078125
89 1031.0875244140625
90 971.4365234375
91 915.4481201171875
92 862.870849609375
93 813.4937133789062
94 767.0994262695312
95 723.5149536132812
96 682.5278930664062
97 643.9902954101562
98 607.7457275390625
99 573.6528930664062
100 541.5621337890625
101 511.3641357421875
102 482.9376525878906
103 456.1539611816406
104 430.923828125
105 407.15985107421875
106 384.7700500488281
107 363.6610107421875
108 343.7568054199219
109 324.99468994140625
110 307.2923583984375
111 290.59735107421875
112 274.8479309082031
113 259.99200439453125
114 245.96633911132812
115 232.7264404296875
116 220.2318572998047
117 208.43003845214844
118 197.2797393798828
119 186.7504425048828
120 176.80433654785156
121 167.40847778320312
122 158.5271759033203
123 150.13357543945312
124 142.19815063476562
125 134.6975860595703
126 127.60556030273438
127 120.89796447753906
128 114.5552978515625
129 108.55418395996094
130 102.87720489501953
131 97.50912475585938
132 92.42718505859375
133 87.61859893798828
134 83.06895446777344
135 78.75951385498047
136 74.68318176269531
137 70.82167053222656
138 67.16414642333984
139 63.70227813720703
140 60.422691345214844
141 57.316871643066406
142 54.37543869018555
143 51.5876350402832
144 48.94669723510742
145 46.44532012939453
146 44.07268142700195
147 41.82688522338867
148 39.69676971435547
149 37.678016662597656
150 35.76398468017578
151 33.94977951049805
152 32.22893524169922
153 30.598487854003906
154 29.050945281982422
155 27.58523178100586
156 26.19336700439453
157 24.87327766418457
158 23.621789932250977
159 22.43433380126953
160 21.30767059326172
161 20.23839569091797
162 19.223857879638672
163 18.261821746826172
164 17.348695755004883
165 16.482309341430664
166 15.658927917480469
167 14.878515243530273
168 14.137420654296875
169 13.43375301361084
170 12.765519142150879
171 12.131643295288086
172 11.529241561889648
173 10.957677841186523
174 10.414859771728516
175 9.899190902709961
176 9.409743309020996
177 8.944790840148926
178 8.50346565246582
179 8.083965301513672
180 7.685507774353027
181 7.307207107543945
182 6.9477128982543945
183 6.606009483337402
184 6.2811689376831055
185 5.97306489944458
186 5.6800713539123535
187 5.4017229080200195
188 5.137021064758301
189 4.885703086853027
190 4.646921157836914
191 4.4199748039245605
192 4.20389986038208
193 3.9990451335906982
194 3.803867816925049
195 3.618668556213379
196 3.4424965381622314
197 3.2748968601226807
198 3.1156654357910156
199 2.9643476009368896
200 2.820277690887451
201 2.683563232421875
202 2.553238868713379
203 2.429581642150879
204 2.31205153465271
205 2.1999940872192383
206 2.0935158729553223
207 1.9925706386566162
208 1.8962925672531128
209 1.8047349452972412
210 1.7176907062530518
211 1.6346653699874878
212 1.5559253692626953
213 1.481032371520996
214 1.4097764492034912
215 1.3420662879943848
216 1.27744460105896
217 1.216089129447937
218 1.1577719449996948
219 1.1021479368209839
220 1.0493303537368774
221 0.9989923238754272
222 0.9512022137641907
223 0.9056214094161987
224 0.8623266816139221
225 0.8209798336029053
226 0.7818542122840881
227 0.7445145845413208
228 0.708891749382019
229 0.6750669479370117
230 0.6429514288902283
231 0.6121802926063538
232 0.5830100178718567
233 0.5553401708602905
234 0.5289539694786072
235 0.5037387609481812
236 0.4797610640525818
237 0.45696163177490234
238 0.43529239296913147
239 0.4146682620048523
240 0.3949935734272003
241 0.37623387575149536
242 0.35838234424591064
243 0.34140297770500183
244 0.3252045512199402
245 0.30983391404151917
246 0.29515567421913147
247 0.2812279462814331
248 0.2679428458213806
249 0.2552989423274994
250 0.243219256401062
251 0.2317039519548416
252 0.2207852005958557
253 0.2103772908449173
254 0.20043915510177612
255 0.1909874528646469
256 0.1820005476474762
257 0.1734415590763092
258 0.16528961062431335
259 0.15749937295913696
260 0.15013322234153748
261 0.14304834604263306
262 0.13633647561073303
263 0.12992756068706512
264 0.12385055422782898
265 0.11803290247917175
266 0.11249157041311264
267 0.10724391788244247
268 0.10220323503017426
269 0.09741441160440445
270 0.09285109490156174
271 0.08850893378257751
272 0.08437427878379822
273 0.0804113894701004
274 0.07667317986488342
275 0.0731005072593689
276 0.06968295574188232
277 0.06644032150506973
278 0.06334859132766724
279 0.06038893386721611
280 0.05757825821638107
281 0.05490751937031746
282 0.05236281082034111
283 0.04991708695888519
284 0.047610774636268616
285 0.04541027545928955
286 0.04329446330666542
287 0.041272833943367004
288 0.03938303515315056
289 0.037569545209407806
290 0.035833973437547684
291 0.03417444974184036
292 0.03258209675550461
293 0.03107640892267227
294 0.02964247390627861
295 0.028284572064876556
296 0.026981541886925697
297 0.02573804184794426
298 0.024550454691052437
299 0.0234342310577631
300 0.022351844236254692
301 0.021333714947104454
302 0.02036045677959919
303 0.01942995935678482
304 0.018537398427724838
305 0.01769576594233513
306 0.01689295656979084
307 0.01612243428826332
308 0.015390120446681976
309 0.014681108295917511
310 0.014016784727573395
311 0.013381056487560272
312 0.012778518721461296
313 0.01219954714179039
314 0.011653874069452286
315 0.011131192557513714
316 0.010627973824739456
317 0.01014798879623413
318 0.009701735340058804
319 0.009269571863114834
320 0.008856071159243584
321 0.008459897711873055
322 0.008088884875178337
323 0.007727767340838909
324 0.00739289540797472
325 0.00706722354516387
326 0.006758130621165037
327 0.006458365358412266
328 0.006173229776322842
329 0.005906475242227316
330 0.005649351514875889
331 0.005397305823862553
332 0.005166457034647465
333 0.004945025779306889
334 0.004737802315503359
335 0.00453278748318553
336 0.004339446779340506
337 0.004149827174842358
338 0.003973860759288073
339 0.003805030370131135
340 0.0036416673101484776
341 0.003491697832942009
342 0.0033451460767537355
343 0.0032032057642936707
344 0.003074738197028637
345 0.0029445053078234196
346 0.002822993090376258
347 0.0027112471871078014
348 0.0025983862578868866
349 0.002494536805897951
350 0.002394086215645075
351 0.0022958035115152597
352 0.0022068985272198915
353 0.00211905175819993
354 0.0020363815128803253
355 0.0019562039524316788
356 0.0018759770318865776
357 0.001802589395083487
358 0.0017334828153252602
359 0.0016670171171426773
360 0.001604120247066021
361 0.0015458270208910108
362 0.0014863302931189537
363 0.0014306183438748121
364 0.0013767085038125515
365 0.0013281517894938588
366 0.00127766455989331
367 0.0012322005350142717
368 0.0011864486150443554
369 0.0011459271190688014
370 0.0011039470555260777
371 0.0010646856389939785
372 0.0010265331948176026
373 0.0009905893821269274
374 0.0009555660653859377
375 0.0009227151749655604
376 0.000891061732545495
377 0.0008600767469033599
378 0.0008304959628731012
379 0.000802570313680917
380 0.0007763823959976435
381 0.0007492949371226132
382 0.0007261931896209717
383 0.0007026303792372346
384 0.0006783809512853622
385 0.0006582411588169634
386 0.0006380325648933649
387 0.0006175294402055442
388 0.0005976230604574084
389 0.0005783046362921596
390 0.0005607994389720261
391 0.0005435555940493941
392 0.0005274737486615777
393 0.0005105417221784592
394 0.0004953040624968708
395 0.00048062813584692776
396 0.00046725000720471144
397 0.00045319218770600855
398 0.00043893407564610243
399 0.0004258319968357682
400 0.00041448994306847453
401 0.0004013273282907903
402 0.0003900786687154323
403 0.00037863629404455423
404 0.0003683574905153364
405 0.00035760330501943827
406 0.000349054200341925
407 0.00033894728403538465
408 0.00032937416108325124
409 0.0003202207153663039
410 0.0003122163179796189
411 0.0003041108138859272
412 0.00029589218320325017
413 0.00028828196809627116
414 0.00028026860672980547
415 0.00027260661590844393
416 0.000265556329395622
417 0.0002589072100818157
418 0.0002519533154554665
419 0.0002457647933624685
420 0.00024008267791941762
421 0.00023445591796189547
422 0.00022757859551347792
423 0.00022183613327797502
424 0.00021664935047738254
425 0.00021192671556491405
426 0.00020677989232353866
427 0.00020107455202378333
428 0.00019645565771497786
429 0.0001925480319187045
430 0.0001878278999356553
431 0.0001835158036556095
432 0.00017846678383648396
433 0.00017447791469749063
434 0.00017066927102860063
435 0.00016691202472429723
436 0.00016320239228662103
437 0.00015947828069329262
438 0.0001558567601023242
439 0.0001523972605355084
440 0.00014944118447601795
441 0.0001461139036109671
442 0.00014254949928727
443 0.00013984231918584555
444 0.00013716117246076465
445 0.0001340295420959592
446 0.00013127701822668314
447 0.00012855725071858615
448 0.00012581582996062934
449 0.00012309220619499683
450 0.00012052943930029869
451 0.0001178809761768207
452 0.00011569385242182761
453 0.00011285995424259454
454 0.000110796740045771
455 0.00010864654905162752
456 0.00010651418415363878
457 0.00010475809540366754
458 0.00010277231194777414
459 0.00010069698328152299
460 9.895439143292606e-05
461 9.693022002466023e-05
462 9.50813337112777e-05
463 9.339681128039956e-05
464 9.145468357019126e-05
465 8.988780609797686e-05
466 8.814372995402664e-05
467 8.6689688032493e-05
468 8.552354847779498e-05
469 8.421471284236759e-05
470 8.29268537927419e-05
471 8.106940367724746e-05
472 7.994837505975738e-05
473 7.837127486709505e-05
474 7.69336984376423e-05
475 7.590449240524322e-05
476 7.463307701982558e-05
477 7.322532474063337e-05
478 7.186461880337447e-05
479 7.070975698297843e-05
480 6.94396803737618e-05
481 6.854320236016065e-05
482 6.734127964591607e-05
483 6.645842222496867e-05
484 6.529381789732724e-05
485 6.419298006221652e-05
486 6.328110612230375e-05
487 6.219915667315945e-05
488 6.149461842142045e-05
489 6.0434704209910706e-05
490 5.9430130932014436e-05
491 5.8492154494160786e-05
492 5.742206849390641e-05
493 5.6744083849480376e-05
494 5.59227482881397e-05
495 5.521724960999563e-05
496 5.4453186749015003e-05
497 5.3943578677717596e-05
498 5.314529698807746e-05
499 5.232082185102627e-05

简单的autograd

# Create tensors.
x = torch.tensor(1., requires_grad=True)
w = torch.tensor(2., requires_grad=True)
b = torch.tensor(3., requires_grad=True)

# Build a computational graph.
y = w * x + b    # y = 2 * x + 3

# Compute gradients.
y.backward()

# Print out the gradients.
print(x.grad)    # x.grad = 2
print(w.grad)    # w.grad = 1
print(b.grad)    # b.grad = 1

tensor(2.)
tensor(1.)
tensor(1.)

PyTorch: Tensor和autograd

PyTorch的一个重要功能就是autograd，也就是说只要定义了forward pass(前向神经网络)，计算了loss之后，PyTorch可以自动求导计算模型所有参数的梯度。

一个PyTorch的Tensor表示计算图中的一个节点。如果x是一个Tensor并且x.requires_grad=True那么x.grad是另一个储存着x当前梯度(相对于一个scalar，常常是loss)的向量。

import torch

dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N 是 batch size; D_in 是 input dimension;
# H 是 hidden dimension; D_out 是 output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# 创建随机的Tensor来保存输入和输出
# 设定requires_grad=False表示在反向传播的时候我们不需要计算gradient
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# 创建随机的Tensor和权重。
# 设置requires_grad=True表示我们希望反向传播的时候计算Tensor的gradient
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    # 前向传播:通过Tensor预测y；这个和普通的神经网络的前向传播没有任何不同，
    # 但是我们不需要保存网络的中间运算结果，因为我们不需要手动计算反向传播。
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    # 通过前向传播计算loss
    # loss是一个形状为(1，)的Tensor
    # loss.item()可以给我们返回一个loss的scalar
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())

    # PyTorch给我们提供了autograd的方法做反向传播。如果一个Tensor的requires_grad=True，
    # backward会自动计算loss相对于每个Tensor的gradient。在backward之后，
    # w1.grad和w2.grad会包含两个loss相对于两个Tensor的gradient信息。
    loss.backward()

    # 我们可以手动做gradient descent(后面我们会介绍自动的方法)。
    # 用torch.no_grad()包含以下statements，因为w1和w2都是requires_grad=True，
    # 但是在更新weights之后我们并不需要再做autograd。
    # 另一种方法是在weight.data和weight.grad.data上做操作，这样就不会对grad产生影响。
    # tensor.data会我们一个tensor，这个tensor和原来的tensor指向相同的内存空间，
    # 但是不会记录计算图的历史。
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

0 27239668.0
1 22798534.0
2 22094932.0
3 21761808.0
4 19912992.0
5 16202208.0
6 11593654.0
7 7523842.5
8 4630695.0
9 2855296.75
10 1833168.0
11 1255533.875
12 919802.8125
13 714099.125
14 579171.375
15 484260.75
16 413177.75
17 357414.78125
18 312135.375
19 274477.9375
20 242646.203125
21 215428.265625
22 191964.65625
23 171609.28125
24 153846.3125
25 138278.34375
26 124578.84375
27 112483.65625
28 101773.71875
29 92269.78125
30 83804.9765625
31 76244.7421875
32 69475.8125
33 63396.06640625
34 57931.9453125
35 53013.28515625
36 48575.20703125
37 44563.7890625
38 40932.0390625
39 37639.265625
40 34645.85546875
41 31923.27734375
42 29444.71484375
43 27184.41796875
44 25119.5078125
45 23231.046875
46 21502.794921875
47 19918.111328125
48 18464.017578125
49 17128.140625
50 15899.7841796875
51 14770.0634765625
52 13732.15625
53 12775.349609375
54 11893.705078125
55 11079.619140625
56 10326.9306640625
57 9630.638671875
58 8985.89453125
59 8388.509765625
60 7834.66064453125
61 7320.845703125
62 6843.84619140625
63 6400.8994140625
64 5989.052734375
65 5606.0224609375
66 5249.70703125
67 4917.85986328125
68 4608.73681640625
69 4320.8818359375
70 4052.4384765625
71 3801.983642578125
72 3568.3046875
73 3350.11572265625
74 3146.266357421875
75 2955.96826171875
76 2778.0302734375
77 2611.5810546875
78 2455.868896484375
79 2310.262939453125
80 2173.9462890625
81 2046.288818359375
82 1926.5927734375
83 1814.3800048828125
84 1709.1483154296875
85 1610.413330078125
86 1517.7894287109375
87 1430.8948974609375
88 1349.2550048828125
89 1272.5718994140625
90 1200.508056640625
91 1132.7694091796875
92 1069.1082763671875
93 1009.2308349609375
94 952.9073486328125
95 899.901611328125
96 850.0130615234375
97 803.0762939453125
98 758.8721923828125
99 717.2393798828125
100 678.0194091796875
101 641.05419921875
102 606.2159423828125
103 573.3717041015625
104 542.396728515625
105 513.189208984375
106 485.63763427734375
107 459.6548767089844
108 435.11163330078125
109 411.946044921875
110 390.08087158203125
111 369.4305725097656
112 349.92999267578125
113 331.5091552734375
114 314.102294921875
115 297.65472412109375
116 282.10272216796875
117 267.4019470214844
118 253.50039672851562
119 240.35340881347656
120 227.9218292236328
121 216.15615844726562
122 205.02935791015625
123 194.4945068359375
124 184.525634765625
125 175.09164428710938
126 166.15489196777344
127 157.69845581054688
128 149.6930694580078
129 142.10499572753906
130 134.91378784179688
131 128.10366821289062
132 121.64862060546875
133 115.53192138671875
134 109.73553466796875
135 104.23783874511719
136 99.02763366699219
137 94.08499145507812
138 89.39903259277344
139 84.9554672241211
140 80.73983001708984
141 76.7408447265625
142 72.94719696044922
143 69.3480224609375
144 65.93144989013672
145 62.69003677368164
146 59.6119384765625
147 56.68977355957031
148 53.917030334472656
149 51.283409118652344
150 48.78217315673828
151 46.41133499145508
152 44.15615463256836
153 42.01270294189453
154 39.97689437866211
155 38.04350280761719
156 36.20542907714844
157 34.45917892456055
158 32.79977798461914
159 31.22269058227539
160 29.7239990234375
161 28.298786163330078
162 26.943511962890625
163 25.65572166442871
164 24.43054962158203
165 23.265382766723633
166 22.157989501953125
167 21.104087829589844
168 20.102310180664062
169 19.148954391479492
170 18.241897583007812
171 17.379301071166992
172 16.558128356933594
173 15.776741981506348
174 15.033305168151855
175 14.32601261138916
176 13.653886795043945
177 13.012701988220215
178 12.402467727661133
179 11.821577072143555
180 11.268671035766602
181 10.742018699645996
182 10.240516662597656
183 9.76303482055664
184 9.308445930480957
185 8.875398635864258
186 8.462831497192383
187 8.070077896118164
188 7.695982933044434
189 7.339229106903076
190 6.999431610107422
191 6.6759114265441895
192 6.367751121520996
193 6.0740885734558105
194 5.794096946716309
195 5.527368545532227
196 5.272972106933594
197 5.030848979949951
198 4.799733638763428
199 4.579549789428711
200 4.369805812835693
201 4.1698150634765625
202 3.9792134761810303
203 3.7975027561187744
204 3.6240322589874268
205 3.4587185382843018
206 3.3010315895080566
207 3.150813341140747
208 3.00724458694458
209 2.8707797527313232
210 2.7401647567749023
211 2.615762233734131
212 2.4972760677337646
213 2.3840444087982178
214 2.2760205268859863
215 2.1732397079467773
216 2.0750210285186768
217 1.981196403503418
218 1.89187753200531
219 1.8064994812011719
220 1.725092887878418
221 1.6474770307540894
222 1.573270320892334
223 1.5025646686553955
224 1.435028076171875
225 1.3706023693084717
226 1.3090440034866333
227 1.2502886056900024
228 1.194399356842041
229 1.140903115272522
230 1.0898479223251343
231 1.0410586595535278
232 0.9946398735046387
233 0.9502640962600708
234 0.9078291654586792
235 0.8673296570777893
236 0.8286425471305847
237 0.7917301654815674
238 0.7564267516136169
239 0.7227426767349243
240 0.6906535029411316
241 0.6599448919296265
242 0.6306719183921814
243 0.6025698184967041
244 0.5758910179138184
245 0.5502774715423584
246 0.5259416103363037
247 0.5026921629905701
248 0.48039209842681885
249 0.459166556596756
250 0.43879884481430054
251 0.41947224736213684
252 0.4009053707122803
253 0.38317784667015076
254 0.36626487970352173
255 0.3501143455505371
256 0.33474063873291016
257 0.3200050890445709
258 0.3058649003505707
259 0.2923714518547058
260 0.27950912714004517
261 0.2672249972820282
262 0.25548774003982544
263 0.24428844451904297
264 0.23355421423912048
265 0.22331443428993225
266 0.21348842978477478
267 0.20414873957633972
268 0.19518357515335083
269 0.1866421401500702
270 0.1784711331129074
271 0.17064383625984192
272 0.1631748229265213
273 0.15604378283023834
274 0.1492093950510025
275 0.14269068837165833
276 0.1364932507276535
277 0.13052017986774445
278 0.12479457259178162
279 0.11935976147651672
280 0.11416064202785492
281 0.10917963087558746
282 0.10444149374961853
283 0.09987418353557587
284 0.09554484486579895
285 0.09138088673353195
286 0.08742237091064453
287 0.08359609544277191
288 0.07995343208312988
289 0.07647553086280823
290 0.07317505776882172
291 0.06998433917760849
292 0.06696024537086487
293 0.06405113637447357
294 0.06127017363905907
295 0.05860717222094536
296 0.056077584624290466
297 0.053649261593818665
298 0.05132327601313591
299 0.04911491274833679
300 0.046983711421489716
301 0.04494979977607727
302 0.0430232509970665
303 0.041141826659440994
304 0.039370082318782806
305 0.03766912221908569
306 0.03605250269174576
307 0.03449958190321922
308 0.033005669713020325
309 0.03158525750041008
310 0.030239086598157883
311 0.028939004987478256
312 0.027689065784215927
313 0.026502318680286407
314 0.025377940386533737
315 0.024286599829792976
316 0.023234477266669273
317 0.02224951796233654
318 0.021299991756677628
319 0.020386626943945885
320 0.0195290669798851
321 0.018697530031204224
322 0.017899617552757263
323 0.017141370102763176
324 0.016418786719441414
325 0.015721982344985008
326 0.01505889743566513
327 0.014414099976420403
328 0.013804873451590538
329 0.013220769353210926
330 0.012667853385210037
331 0.012133610434830189
332 0.01162355113774538
333 0.011125347577035427
334 0.010666405782103539
335 0.010222582146525383
336 0.009801669046282768
337 0.009391330182552338
338 0.008997078984975815
339 0.008630824275314808
340 0.008272996172308922
341 0.007930200546979904
342 0.007602583151310682
343 0.007287715096026659
344 0.006990152411162853
345 0.006704687140882015
346 0.006430609151721001
347 0.006167884450405836
348 0.00592105183750391
349 0.005677740555256605
350 0.005448729731142521
351 0.005227771122008562
352 0.005015998147428036
353 0.004816721193492413
354 0.00462984386831522
355 0.0044435011222958565
356 0.0042654974386096
357 0.004097559954971075
358 0.003936645574867725
359 0.0037810830399394035
360 0.0036363208200782537
361 0.0034961989149451256
362 0.003362423274666071
363 0.0032338243909180164
364 0.0031072921119630337
365 0.002988876309245825
366 0.0028757965192198753
367 0.0027693076990544796
368 0.002663201419636607
369 0.002562501933425665
370 0.002467155922204256
371 0.002376919612288475
372 0.002286899136379361
373 0.002204512245953083
374 0.0021254979074001312
375 0.0020460346713662148
376 0.001973430858924985
377 0.0019028272945433855
378 0.001835328177548945
379 0.0017705523641780019
380 0.0017069694586098194
381 0.0016470919363200665
382 0.0015892802039161325
383 0.0015330100432038307
384 0.001482617575675249
385 0.0014297992456704378
386 0.0013813500991091132
387 0.001332945772446692
388 0.0012858662521466613
389 0.0012436110991984606
390 0.0012005207827314734
391 0.0011599815916270018
392 0.0011221729218959808
393 0.0010862643830478191
394 0.001048326725140214
395 0.0010145889827981591
396 0.000982651486992836
397 0.0009503097971901298
398 0.0009206897229887545
399 0.000892468320671469
400 0.0008637488353997469
401 0.0008374633034691215
402 0.0008101868443191051
403 0.0007849236717447639
404 0.0007609085878357291
405 0.0007373671978712082
406 0.0007157273939810693
407 0.0006949678645469248
408 0.0006736431387253106
409 0.000655067153275013
410 0.0006351302145048976
411 0.0006163989310152829
412 0.0005990251083858311
413 0.0005810378934256732
414 0.0005648763035424054
415 0.0005476527148857713
416 0.0005323370569385588
417 0.0005175815895199776
418 0.0005039768293499947
419 0.000490540754981339
420 0.0004750383086502552
421 0.0004635825753211975
422 0.0004508931306190789
423 0.00043917988659814
424 0.0004265513562131673
425 0.0004155129718128592
426 0.0004043789813295007
427 0.00039464770816266537
428 0.0003840273420792073
429 0.00037375115789473057
430 0.0003653681487776339
431 0.00035570785985328257
432 0.00034683255944401026
433 0.00033835662179626524
434 0.0003299047239124775
435 0.00032125471625477076
436 0.0003139039035886526
437 0.00030607334338128567
438 0.00029867544071748853
439 0.0002918266982305795
440 0.0002847398864105344
441 0.00027804437559098005
442 0.0002716465387493372
443 0.00026566567248664796
444 0.0002598784340079874
445 0.0002526362077333033
446 0.0002473775530233979
447 0.00024159630993381143
448 0.00023634321405552328
449 0.00023139714903663844
450 0.00022599002113565803
451 0.00022058063768781722
452 0.00021629821276292205
453 0.00021151063265278935
454 0.00020658751600421965
455 0.0002023529086727649
456 0.00019809410150628537
457 0.00019375921692699194
458 0.0001897729089250788
459 0.0001857166353147477
460 0.00018225228996016085
461 0.0001781475730240345
462 0.00017454184126108885
463 0.00017124674923252314
464 0.000167444595717825
465 0.00016409793170168996
466 0.00016078155022114515
467 0.00015773865743540227
468 0.00015477021224796772
469 0.00015141766925808042
470 0.00014836443006061018
471 0.0001458464830648154
472 0.00014257020666263998
473 0.00014018468209542334
474 0.00013717498222831637
475 0.00013514941383618861
476 0.00013247963215690106
477 0.00012970241368748248
478 0.0001271112123504281
479 0.00012479811266530305
480 0.00012312702892813832
481 0.00012058767606504261
482 0.00011860761151183397
483 0.00011626520426943898
484 0.00011402984091546386
485 0.00011233107215957716
486 0.00011016873031621799
487 0.0001086159172700718
488 0.00010658100654836744
489 0.00010471833229530603
490 0.00010301855218131095
491 0.00010130059672519565
492 9.948517981683835e-05
493 9.802125714486465e-05
494 9.652116568759084e-05
495 9.46744839893654e-05
496 9.31027316255495e-05
497 9.148970275418833e-05
498 8.99480510270223e-05
499 8.85994013515301e-05

PyTorch: nn

这次我们使用PyTorch中nn这个库来构建网络。
用PyTorch autograd来构建计算图和计算gradients，
然后PyTorch会帮我们自动计算gradient。

import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Use the nn package to define our model as a sequence of layers. nn.Sequential
# is a Module which contains other Modules, and applies them in sequence to
# produce its output. Each Linear Module computes output from input using a
# linear function, and holds internal Tensors for its weight and bias.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)

# The nn package also contains definitions of popular loss functions; in this
# case we will use Mean Squared Error (MSE) as our loss function.
loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(500):
    # Forward pass: compute predicted y by passing x to the model. Module objects
    # override the __call__ operator so you can call them like functions. When
    # doing so you pass a Tensor of input data to the Module and it produces
    # a Tensor of output data.
    y_pred = model(x)

    # Compute and print loss. We pass Tensors containing the predicted and true
    # values of y, and the loss function returns a Tensor containing the
    # loss.
    loss = loss_fn(y_pred, y)
    print(t, loss.item())

    # Zero the gradients before running the backward pass.
    model.zero_grad()

    # Backward pass: compute gradient of the loss with respect to all the learnable
    # parameters of the model. Internally, the parameters of each Module are stored
    # in Tensors with requires_grad=True, so this call will compute gradients for
    # all learnable parameters in the model.
    loss.backward()

    # Update the weights using gradient descent. Each parameter is a Tensor, so
    # we can access its gradients like we did before.
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad

0 730.4003295898438
1 677.806884765625
2 632.0365600585938
3 591.8120727539062
4 555.9887084960938
5 523.8406372070312
6 494.6942443847656
7 467.8385925292969
8 442.7820129394531
9 419.5081787109375
10 397.74688720703125
11 377.21978759765625
12 357.9049987792969
13 339.6867980957031
14 322.4093933105469
15 305.97412109375
16 290.2826232910156
17 275.3173522949219
18 261.0522766113281
19 247.37789916992188
20 234.25460815429688
21 221.73854064941406
22 209.75025939941406
23 198.29336547851562
24 187.3721160888672
25 176.96902465820312
26 167.0499725341797
27 157.61865234375
28 148.6443328857422
29 140.12136840820312
30 132.05642700195312
31 124.41332244873047
32 117.17528533935547
33 110.3093490600586
34 103.81737518310547
35 97.68974304199219
36 91.9076919555664
37 86.44802856445312
38 81.30424499511719
39 76.45010375976562
40 71.87824249267578
41 67.57888793945312
42 63.540672302246094
43 59.74314498901367
44 56.17317199707031
45 52.8209228515625
46 49.66929626464844
47 46.71091842651367
48 43.93756103515625
49 41.3336067199707
50 38.89524841308594
51 36.60483169555664
52 34.46003723144531
53 32.442466735839844
54 30.55179786682129
55 28.779720306396484
56 27.11409568786621
57 25.54958724975586
58 24.080965042114258
59 22.7009220123291
60 21.406597137451172
61 20.18861198425293
62 19.04090118408203
63 17.963912963867188
64 16.9540958404541
65 16.00415802001953
66 15.109206199645996
67 14.260186195373535
68 13.463393211364746
69 12.714924812316895
70 12.01164722442627
71 11.349655151367188
72 10.727004051208496
73 10.141312599182129
74 9.591001510620117
75 9.072677612304688
76 8.583186149597168
77 8.12155532836914
78 7.687155723571777
79 7.277516841888428
80 6.891292572021484
81 6.527829170227051
82 6.185011863708496
83 5.861148834228516
84 5.555330276489258
85 5.266604423522949
86 4.994298934936523
87 4.737382888793945
88 4.494537353515625
89 4.264896392822266
90 4.047701358795166
91 3.842176914215088
92 3.6478099822998047
93 3.4641051292419434
94 3.290283679962158
95 3.1227707862854004
96 2.9646613597869873
97 2.815171003341675
98 2.6738247871398926
99 2.540189266204834
100 2.413630723953247
101 2.293856382369995
102 2.180544853210449
103 2.0732321739196777
104 1.9715577363967896
105 1.8752238750457764
106 1.7838845252990723
107 1.6973004341125488
108 1.6151288747787476
109 1.5372742414474487
110 1.4634958505630493
111 1.393602967262268
112 1.327257513999939
113 1.264251708984375
114 1.204415202140808
115 1.1476118564605713
116 1.0936859846115112
117 1.0424509048461914
118 0.9937999248504639
119 0.9473934769630432
120 0.9033433198928833
121 0.861416757106781
122 0.8215345740318298
123 0.7836579084396362
124 0.747665286064148
125 0.7134326696395874
126 0.6808668375015259
127 0.6498791575431824
128 0.6203745603561401
129 0.5922937989234924
130 0.5656096339225769
131 0.5402023792266846
132 0.5159667730331421
133 0.4928816556930542
134 0.4708971679210663
135 0.4499624967575073
136 0.42999905347824097
137 0.41098135709762573
138 0.39284196496009827
139 0.3755357265472412
140 0.3590378761291504
141 0.34330621361732483
142 0.3283001482486725
143 0.3139764368534088
144 0.3003006875514984
145 0.28725573420524597
146 0.27479085326194763
147 0.26290929317474365
148 0.25157156586647034
149 0.24073074758052826
150 0.23037873208522797
151 0.22049961984157562
152 0.21105742454528809
153 0.20203885436058044
154 0.19341179728507996
155 0.18518121540546417
156 0.17731204628944397
157 0.16979391872882843
158 0.16260133683681488
159 0.1557299643754959
160 0.1491609662771225
161 0.14287640154361725
162 0.1368718445301056
163 0.13112519681453705
164 0.1256367713212967
165 0.12037747353315353
166 0.11534851044416428
167 0.11053863167762756
168 0.10593828558921814
169 0.10153594613075256
170 0.09732852876186371
171 0.09330316632986069
172 0.08944696933031082
173 0.08575168997049332
174 0.08221769332885742
175 0.07883680611848831
176 0.0755985900759697
177 0.07249889522790909
178 0.06953243911266327
179 0.06669127196073532
180 0.0639720931649208
181 0.06137032061815262
182 0.05887451767921448
183 0.05648396164178848
184 0.05419414862990379
185 0.052000612020492554
186 0.049897633492946625
187 0.04788323864340782
188 0.045953795313835144
189 0.044105008244514465
190 0.04233371466398239
191 0.040637414902448654
192 0.039010848850011826
193 0.03745134547352791
194 0.03595568612217903
195 0.03452153131365776
196 0.03314797207713127
197 0.03183085843920708
198 0.03056834265589714
199 0.029356608167290688
200 0.02819407545030117
201 0.027081340551376343
202 0.026012500748038292
203 0.02498689666390419
204 0.024003855884075165
205 0.023060820996761322
206 0.02215633913874626
207 0.02128800004720688
208 0.020454630255699158
209 0.019655294716358185
210 0.018888482823967934
211 0.01815379038453102
212 0.01744757406413555
213 0.01676984876394272
214 0.016118871048092842
215 0.015494021587073803
216 0.014894749969244003
217 0.014319296926259995
218 0.01376650296151638
219 0.013235821388661861
220 0.012726814486086369
221 0.012238302268087864
222 0.011768481694161892
223 0.011317926459014416
224 0.010885199531912804
225 0.010469462722539902
226 0.010070198215544224
227 0.00968638714402914
228 0.009317745454609394
229 0.008963854052126408
230 0.008623997680842876
231 0.008297951892018318
232 0.007983983494341373
233 0.007682369090616703
234 0.00739262392744422
235 0.007114121690392494
236 0.00684656947851181
237 0.00658929068595171
238 0.0063418918289244175
239 0.006104208994656801
240 0.005875867325812578
241 0.005656401161104441
242 0.005445381626486778
243 0.005242596380412579
244 0.005047555081546307
245 0.004859848879277706
246 0.004679631907492876
247 0.004506140481680632
248 0.004339287523180246
249 0.004178992938250303
250 0.0040248180739581585
251 0.0038765515200793743
252 0.0037338845431804657
253 0.0035966155119240284
254 0.0034651204477995634
255 0.0033386265859007835
256 0.0032169343903660774
257 0.0030999258160591125
258 0.0029872951563447714
259 0.0028788733761757612
260 0.002774540102109313
261 0.002674160525202751
262 0.0025775693356990814
263 0.002484537661075592
264 0.002395007759332657
265 0.002308792434632778
266 0.0022258535027503967
267 0.0021460168063640594
268 0.00206911307759583
269 0.0019950829446315765
270 0.0019238003296777606
271 0.0018551491666585207
272 0.001789016299881041
273 0.0017253642436116934
274 0.0016640406101942062
275 0.0016050608828663826
276 0.001548313070088625
277 0.0014935110229998827
278 0.0014407128328457475
279 0.0013899386394768953
280 0.0013409735402092338
281 0.0012938103172928095
282 0.0012483753962442279
283 0.0012045733164995909
284 0.0011623809114098549
285 0.0011217109858989716
286 0.0010825133649632335
287 0.0010447640670463443
288 0.0010083483066409826
289 0.0009732777252793312
290 0.0009394678054377437
291 0.0009068790823221207
292 0.0008754512527957559
293 0.0008451729663647711
294 0.0008159802528098226
295 0.0007878387696109712
296 0.0007607016596011817
297 0.0007345342892222106
298 0.0007092973100952804
299 0.0006849935743957758
300 0.0006615345482714474
301 0.0006388957845047116
302 0.0006170707056298852
303 0.0005960320122539997
304 0.0005757393664680421
305 0.000556156795937568
306 0.000537263578735292
307 0.0005190335214138031
308 0.0005014348425902426
309 0.00048447478911839426
310 0.0004681102291215211
311 0.0004523076640907675
312 0.00043707736767828465
313 0.00042236282024532557
314 0.0004081761871930212
315 0.0003944859199691564
316 0.00038126655272208154
317 0.0003685199189931154
318 0.00035620472044683993
319 0.00034432471147738397
320 0.00033285238896496594
321 0.0003217716293875128
322 0.0003110812685918063
323 0.00030076270923018456
324 0.00029080003150738776
325 0.0002811678859870881
326 0.0002718767209444195
327 0.00026289874222129583
328 0.0002542325237300247
329 0.0002458583330735564
330 0.00023778063768986613
331 0.0002299778861925006
332 0.00022243551211431623
333 0.0002151544758817181
334 0.00020812121510971338
335 0.00020132500503677875
336 0.00019475606677588075
337 0.00018840879783965647
338 0.00018227536929771304
339 0.0001763583131833002
340 0.00017063299310393631
341 0.0001651006896281615
342 0.00015975921996869147
343 0.00015459131100215018
344 0.00014960448606871068
345 0.00014478273806162179
346 0.0001401226472808048
347 0.00013561020023189485
348 0.00013125850819051266
349 0.0001270477077923715
350 0.00012297916691750288
351 0.00011904665007023141
352 0.00011524671572260559
353 0.00011156262189615518
354 0.00010800941527122632
355 0.00010456875315867364
356 0.00010124015534529462
357 9.802745626075193e-05
358 9.49160530581139e-05
359 9.191099525196478e-05
360 8.900299872038886e-05
361 8.619319123681635e-05
362 8.346916001755744e-05
363 8.083278953563422e-05
364 7.829233072698116e-05
365 7.582692342111841e-05
366 7.344653567997739e-05
367 7.114354957593605e-05
368 6.891170050948858e-05
369 6.675738404737785e-05
370 6.466719059972093e-05
371 6.264891271712258e-05
372 6.069618757464923e-05
373 5.880574099137448e-05
374 5.6973563914652914e-05
375 5.520202103070915e-05
376 5.348709601094015e-05
377 5.182927270652726e-05
378 5.0222464778926224e-05
379 4.866871677222662e-05
380 4.716542025562376e-05
381 4.570980308926664e-05
382 4.4300137233221903e-05
383 4.29377869295422e-05
384 4.161441029282287e-05
385 4.033445657114498e-05
386 3.909708539140411e-05
387 3.789710899582133e-05
388 3.673461469588801e-05
389 3.561353150871582e-05
390 3.4523382055340335e-05
391 3.346895391587168e-05
392 3.244729668949731e-05
393 3.145977098029107e-05
394 3.0503028028761037e-05
395 2.9576976885437034e-05
396 2.867497823899612e-05
397 2.7807111109723337e-05
398 2.6963021809933707e-05
399 2.6149191398872063e-05
400 2.5358009224873967e-05
401 2.4591785404481925e-05
402 2.384811705269385e-05
403 2.313069489900954e-05
404 2.2434744096244685e-05
405 2.1759187802672386e-05
406 2.1105141058797017e-05
407 2.047013731498737e-05
408 1.9857605366269127e-05
409 1.9260078261140734e-05
410 1.8684197129914537e-05
411 1.8125667338608764e-05
412 1.7582991858944297e-05
413 1.705808608676307e-05
414 1.6549734937143512e-05
415 1.6056841559475288e-05
416 1.5579344108118676e-05
417 1.5116088434297126e-05
418 1.4666821698483545e-05
419 1.4232056855689734e-05
420 1.3808910807711072e-05
421 1.339945083600469e-05
422 1.3002342711843085e-05
423 1.2619992048712447e-05
424 1.2246448022779077e-05
425 1.1884645573445596e-05
426 1.1535368685144931e-05
427 1.1194167200301308e-05
428 1.086501288227737e-05
429 1.054473250405863e-05
430 1.0235738955088891e-05
431 9.936196875059977e-06
432 9.643772500567138e-06
433 9.36158812692156e-06
434 9.087586477107834e-06
435 8.820732546155341e-06
436 8.563694791519083e-06
437 8.31347188068321e-06
438 8.071336196735501e-06
439 7.83547329774592e-06
440 7.607360657857498e-06
441 7.384997843473684e-06
442 7.172181085479679e-06
443 6.962596216908423e-06
444 6.760796168236993e-06
445 6.564631803485099e-06
446 6.374919394147582e-06
447 6.1895607359474525e-06
448 6.011287496221485e-06
449 5.837395292473957e-06
450 5.668822268489748e-06
451 5.504605269379681e-06
452 5.3455041779670864e-06
453 5.191865056985989e-06
454 5.043272267357679e-06
455 4.897161488770507e-06
456 4.756713224196574e-06
457 4.620327217708109e-06
458 4.487657861318439e-06
459 4.358199475973379e-06
460 4.233525487506995e-06
461 4.111566340725403e-06
462 3.994332018919522e-06
463 3.880317763105268e-06
464 3.7690213048335863e-06
465 3.66149538422178e-06
466 3.556691353878705e-06
467 3.456340891716536e-06
468 3.3571118365216535e-06
469 3.261705614931998e-06
470 3.1689476145402296e-06
471 3.0789522043050965e-06
472 2.9916900530224666e-06
473 2.9056911898805993e-06
474 2.8240870051376987e-06
475 2.7439589302957756e-06
476 2.6666020858101547e-06
477 2.5906620066962205e-06
478 2.5173019366775407e-06
479 2.4466010017931694e-06
480 2.3770278403389966e-06
481 2.309647243237123e-06
482 2.2444771730079083e-06
483 2.1814687443111325e-06
484 2.1197672595008044e-06
485 2.0598859009623993e-06
486 2.001977918553166e-06
487 1.9454662378848298e-06
488 1.891090164463094e-06
489 1.8379661241851863e-06
490 1.7857651073427405e-06
491 1.7361446680297377e-06
492 1.6874179209480644e-06
493 1.6402980236307485e-06
494 1.59431692736689e-06
495 1.549733156025468e-06
496 1.5061962130857864e-06
497 1.464243382542918e-06
498 1.4235124581318814e-06
499 1.3836001926392782e-06

PyTorch: optim

这一次我们不再手动更新模型的weights,而是使用optim这个包来帮助我们更新参数。
optim这个package提供了各种不同的模型优化方法，包括SGD+momentum, RMSProp, Adam等等。

import torch

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
    torch.nn.Linear(D_in, H),
    torch.nn.ReLU(),
    torch.nn.Linear(H, D_out),
)
loss_fn = torch.nn.MSELoss(reduction='sum')

# Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use Adam; the optim package contains many other
# optimization algoriths. The first argument to the Adam constructor tells the
# optimizer which Tensors it should update.
learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for t in range(500):
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(x)

    # Compute and print loss.
    loss = loss_fn(y_pred, y)
    print(t, loss.item())

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable
    # weights of the model). This is because by default, gradients are
    # accumulated in buffers( i.e, not overwritten) whenever .backward()
    # is called. Checkout docs of torch.autograd.backward for more details.
    optimizer.zero_grad()

    # Backward pass: compute gradient of the loss with respect to model
    # parameters
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()

0 689.9976806640625
1 673.0743408203125
2 656.5448608398438
3 640.480224609375
4 624.8812255859375
5 609.6969604492188
6 594.9974975585938
7 580.7254638671875
8 566.7864379882812
9 553.24755859375
10 540.17919921875
11 527.5197143554688
12 515.1770629882812
13 503.2459716796875
14 491.66583251953125
15 480.41650390625
16 469.43304443359375
17 458.7113952636719
18 448.26031494140625
19 438.1050109863281
20 428.2442321777344
21 418.6424255371094
22 409.2586975097656
23 400.0559997558594
24 391.06378173828125
25 382.2933044433594
26 373.7290954589844
27 365.3452453613281
28 357.1343994140625
29 349.11395263671875
30 341.2725524902344
31 333.6096496582031
32 326.1455383300781
33 318.8943786621094
34 311.79296875
35 304.8321228027344
36 298.00421142578125
37 291.3017578125
38 284.72796630859375
39 278.2953796386719
40 271.98834228515625
41 265.80108642578125
42 259.7551574707031
43 253.81771850585938
44 247.9820098876953
45 242.27828979492188
46 236.71278381347656
47 231.25462341308594
48 225.90187072753906
49 220.6475372314453
50 215.49343872070312
51 210.4272003173828
52 205.45330810546875
53 200.56578063964844
54 195.7642059326172
55 191.05291748046875
56 186.44845581054688
57 181.92417907714844
58 177.49209594726562
59 173.14093017578125
60 168.8763427734375
61 164.67874145507812
62 160.56561279296875
63 156.5225372314453
64 152.5622100830078
65 148.67532348632812
66 144.8670654296875
67 141.1275634765625
68 137.45455932617188
69 133.86842346191406
70 130.3537139892578
71 126.90625
72 123.53035736083984
73 120.21866607666016
74 116.97648620605469
75 113.804931640625
76 110.69839477539062
77 107.65438842773438
78 104.67524719238281
79 101.75517272949219
80 98.9085464477539
81 96.1213607788086
82 93.40021514892578
83 90.73613739013672
84 88.12960052490234
85 85.58183288574219
86 83.08662414550781
87 80.65141296386719
88 78.2701416015625
89 75.9460678100586
90 73.6756362915039
91 71.45594024658203
92 69.2869873046875
93 67.17097473144531
94 65.10391235351562
95 63.08529281616211
96 61.1196403503418
97 59.204254150390625
98 57.332706451416016
99 55.510047912597656
100 53.7337646484375
101 52.00457000732422
102 50.32087707519531
103 48.68069076538086
104 47.08634948730469
105 45.53308868408203
106 44.02242660522461
107 42.5555534362793
108 41.13151550292969
109 39.74758529663086
110 38.40310287475586
111 37.09666442871094
112 35.827293395996094
113 34.593162536621094
114 33.39449691772461
115 32.231807708740234
116 31.100849151611328
117 30.003196716308594
118 28.939111709594727
119 27.907108306884766
120 26.904752731323242
121 25.93457794189453
122 24.995201110839844
123 24.08201789855957
124 23.198774337768555
125 22.34354019165039
126 21.514095306396484
127 20.711238861083984
128 19.932661056518555
129 19.181987762451172
130 18.454797744750977
131 17.75270652770996
132 17.07206153869629
133 16.414682388305664
134 15.779529571533203
135 15.16641902923584
136 14.574522018432617
137 14.003190040588379
138 13.451677322387695
139 12.919203758239746
140 12.405925750732422
141 11.91085147857666
142 11.433086395263672
143 10.972294807434082
144 10.527291297912598
145 10.098633766174316
146 9.685166358947754
147 9.286446571350098
148 8.902650833129883
149 8.533836364746094
150 8.178427696228027
151 7.835862636566162
152 7.506545066833496
153 7.18995475769043
154 6.885422229766846
155 6.592419624328613
156 6.310509204864502
157 6.040183067321777
158 5.7797064781188965
159 5.529158592224121
160 5.28851842880249
161 5.057610988616943
162 4.835738182067871
163 4.622567176818848
164 4.418156147003174
165 4.221595764160156
166 4.033288955688477
167 3.852569341659546
168 3.679222583770752
169 3.5130491256713867
170 3.353869676589966
171 3.2011170387268066
172 3.054814577102661
173 2.914799690246582
174 2.7809627056121826
175 2.6523687839508057
176 2.5295581817626953
177 2.412156581878662
178 2.299792528152466
179 2.1923320293426514
180 2.0895402431488037
181 1.9913344383239746
182 1.8974617719650269
183 1.8077517747879028
184 1.7221109867095947
185 1.6402111053466797
186 1.5621683597564697
187 1.4876220226287842
188 1.4162794351577759
189 1.348184585571289
190 1.2832691669464111
191 1.2212234735488892
192 1.1621438264846802
193 1.1056087017059326
194 1.0518066883087158
195 1.0004193782806396
196 0.9514893293380737
197 0.9047578573226929
198 0.8602332472801208
199 0.817760169506073
200 0.7773001790046692
201 0.7387447357177734
202 0.7020529508590698
203 0.667029619216919
204 0.6336787939071655
205 0.6019449830055237
206 0.5716989040374756
207 0.5429436564445496
208 0.5154978632926941
209 0.48941412568092346
210 0.46459341049194336
211 0.44097888469696045
212 0.41850540041923523
213 0.397123247385025
214 0.37678423523902893
215 0.3574730157852173
216 0.33907800912857056
217 0.32159891724586487
218 0.30498990416526794
219 0.2892071604728699
220 0.2742026448249817
221 0.2599371373653412
222 0.24639734625816345
223 0.23351413011550903
224 0.2212778627872467
225 0.20965951681137085
226 0.19862617552280426
227 0.18814422190189362
228 0.17820759117603302
229 0.16877911984920502
230 0.1598283052444458
231 0.15133057534694672
232 0.14326877892017365
233 0.13563105463981628
234 0.1283716857433319
235 0.12149547040462494
236 0.11496801674365997
237 0.1087847650051117
238 0.10292857885360718
239 0.09737732261419296
240 0.0921185165643692
241 0.08713071048259735
242 0.08240596950054169
243 0.0779285803437233
244 0.07368862628936768
245 0.06966885179281235
246 0.06586882472038269
247 0.06226218119263649
248 0.0588555708527565
249 0.0556354895234108
250 0.052587930113077164
251 0.04970229044556618
252 0.04697170481085777
253 0.044386494904756546
254 0.04194127395749092
255 0.03962467610836029
256 0.03743525967001915
257 0.03536558151245117
258 0.033404912799596786
259 0.03155055642127991
260 0.029797306284308434
261 0.028139181435108185
262 0.026570118963718414
263 0.025088194757699966
264 0.023684784770011902
265 0.022358963266015053
266 0.021105553954839706
267 0.019921932369470596
268 0.018801938742399216
269 0.017742976546287537
270 0.01674332097172737
271 0.01579819619655609
272 0.014904794283211231
273 0.014061586931347847
274 0.013263880275189877
275 0.012512494809925556
276 0.011799972504377365
277 0.011128203012049198
278 0.010494370013475418
279 0.009894886054098606
280 0.009329025633633137
281 0.008794576860964298
282 0.00829020980745554
283 0.007814357988536358
284 0.0073651778511703014
285 0.006940783467143774
286 0.006540366914123297
287 0.00616262573748827
288 0.0058061303570866585
289 0.005469807889312506
290 0.005152509547770023
291 0.004853196442127228
292 0.004570807330310345
293 0.004304558504372835
294 0.004053488373756409
295 0.0038165778387337923
296 0.0035932702012360096
297 0.0033827009610831738
298 0.003184266621246934
299 0.002997200470417738
300 0.0028208219446241856
301 0.0026546064764261246
302 0.0024979272857308388
303 0.002350295428186655
304 0.002211210085079074
305 0.0020801795180886984
306 0.001956697553396225
307 0.001840373151935637
308 0.0017307971138507128
309 0.001627662917599082
310 0.0015304764965549111
311 0.0014389384305104613
312 0.0013527413830161095
313 0.0012715983903035522
314 0.0011951818596571684
315 0.0011232855031266809
316 0.0010555974440649152
317 0.0009919062722474337
318 0.0009319376549683511
319 0.0008755010203458369
320 0.0008224441553466022
321 0.0007724667666479945
322 0.0007254838128574193
323 0.0006812879582867026
324 0.000639721576590091
325 0.0006006223848089576
326 0.0005638641305267811
327 0.0005292984424158931
328 0.0004967894637957215
329 0.00046623218804597855
330 0.00043751858174800873
331 0.0004105096450075507
332 0.0003851380606647581
333 0.0003613029548432678
334 0.00033888546749949455
335 0.000317850528517738
336 0.00029807182727381587
337 0.00027949613286182284
338 0.00026205365429632366
339 0.0002456636866554618
340 0.00023026271082926542
341 0.00021582211775239557
342 0.00020226274500600994
343 0.00018952247046399862
344 0.00017756687884684652
345 0.00016633959603495896
346 0.00015580740000586957
347 0.00014593316882383078
348 0.00013666533050127327
349 0.00012796788359992206
350 0.00011981032730545849
351 0.00011215509584872052
352 0.00010498267511138692
353 9.825362940318882e-05
354 9.194377344101667e-05
355 8.602937305113301e-05
356 8.047981100389734e-05
357 7.528548303525895e-05
358 7.041687786113471e-05
359 6.58537755953148e-05
360 6.157113239169121e-05
361 5.7568802731111646e-05
362 5.381255687098019e-05
363 5.0295715482207015e-05
364 4.700351564679295e-05
365 4.392474511405453e-05
366 4.103749233763665e-05
367 3.833707523881458e-05
368 3.5808108805213124e-05
369 3.344217475387268e-05
370 3.1224517442751676e-05
371 2.915325057983864e-05
372 2.7215797672397457e-05
373 2.5399718651897274e-05
374 2.370383299421519e-05
375 2.2119305867818184e-05
376 2.0636745830415748e-05
377 1.925227843457833e-05
378 1.795551725081168e-05
379 1.674638406257145e-05
380 1.5612897186656483e-05
381 1.4557422218786087e-05
382 1.3567963833338581e-05
383 1.2645687093026936e-05
384 1.1783406080212444e-05
385 1.0980811566696502e-05
386 1.022770538838813e-05
387 9.526746907795314e-06
388 8.871407771948725e-06
389 8.261343282356393e-06
390 7.690169695706572e-06
391 7.158080279623391e-06
392 6.661492989223916e-06
393 6.200371899467427e-06
394 5.767282800661633e-06
395 5.364493972592754e-06
396 4.989693934476236e-06
397 4.639853159460472e-06
398 4.3142304093635175e-06
399 4.010501015727641e-06
400 3.7271922792569967e-06
401 3.4641554975678446e-06
402 3.2186644602916203e-06
403 2.9893835744587705e-06
404 2.777238933049375e-06
405 2.5794286102609476e-06
406 2.3954437438078457e-06
407 2.2237340999708977e-06
408 2.064158024950302e-06
409 1.9158069335389882e-06
410 1.7780365624275873e-06
411 1.64914786182635e-06
412 1.5303024838431156e-06
413 1.4193302604326163e-06
414 1.316024167863361e-06
415 1.2196958323329454e-06
416 1.1306759688523016e-06
417 1.0477258456376148e-06
418 9.710846597954514e-07
419 8.997395752885495e-07
420 8.332587526638235e-07
421 7.720501571384375e-07
422 7.146346661102143e-07
423 6.611753065044468e-07
424 6.125081881691585e-07
425 5.668887297360925e-07
426 5.245773877504689e-07
427 4.851949029216485e-07
428 4.4877231175632915e-07
429 4.149950711962447e-07
430 3.83764785283347e-07
431 3.546888933669834e-07
432 3.2808591754474037e-07
433 3.029143726962502e-07
434 2.802064216211875e-07
435 2.5871841558000597e-07
436 2.390443398780917e-07
437 2.2079815664710623e-07
438 2.0381911269851116e-07
439 1.88296297665147e-07
440 1.7362332016546134e-07
441 1.6036206318403856e-07
442 1.4798970937590639e-07
443 1.3654228325776785e-07
444 1.2592226994456723e-07
445 1.1614223893730014e-07
446 1.0700595254320433e-07
447 9.859136440582006e-08
448 9.087515451255967e-08
449 8.384845529008089e-08
450 7.721519779124719e-08
451 7.12827272764116e-08
452 6.561729293252938e-08
453 6.041440059334491e-08
454 5.558959870199942e-08
455 5.123945001628272e-08
456 4.7194006924655696e-08
457 4.3466727106533654e-08
458 3.991254615698381e-08
459 3.685293137323242e-08
460 3.38908741071009e-08
461 3.118933022960846e-08
462 2.861895964656469e-08
463 2.6406166142578513e-08
464 2.4266151754659404e-08
465 2.2339772698387605e-08
466 2.0516120358138323e-08
467 1.8907332588469217e-08
468 1.7402557617174352e-08
469 1.597093479688283e-08
470 1.4659770286584717e-08
471 1.3471789017671654e-08
472 1.2411474514806287e-08
473 1.1411766642765997e-08
474 1.0501155500719506e-08
475 9.625312102912176e-09
476 8.890474134659598e-09
477 8.151414654378186e-09
478 7.52521156499597e-09
479 6.93865143119865e-09
480 6.397653073975107e-09
481 5.912824452281029e-09
482 5.462171159820173e-09
483 4.996821179048538e-09
484 4.611044879254678e-09
485 4.253352781091735e-09
486 3.934784498227373e-09
487 3.6688359017489347e-09
488 3.3671772037280334e-09
489 3.1128357669274465e-09
490 2.8949134200928484e-09
491 2.662502884831497e-09
492 2.4788016084187348e-09
493 2.308401025885587e-09
494 2.1372417169374103e-09
495 1.9903807491061798e-09
496 1.8517727351508029e-09
497 1.7326815537899165e-09
498 1.6050468731876322e-09
499 1.491649026519326e-09

PyTorch: 自定义 nn Modules

我们可以定义一个模型，这个模型继承自nn.Module类。如果需要定义一个比Sequential模型更加复杂的模型，就需要定义nn.Module模型。

import torch


class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred


# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random Tensors to hold inputs and outputs
x = torch.randn(N, D_in)
y = torch.randn(N, D_out)

# Construct our model by instantiating the class defined above
model = TwoLayerNet(D_in, H, D_out)

# Construct our loss function and an Optimizer. The call to model.parameters()
# in the SGD constructor will contain the learnable parameters of the two
# nn.Linear modules which are members of the model.
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-4)
for t in range(500):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x)

    # Compute and print loss
    loss = criterion(y_pred, y)
    print(t, loss.item())

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

0 650.2210083007812
1 601.2785034179688
2 559.1675415039062
3 522.359130859375
4 489.7039489746094
5 460.2625732421875
6 433.5502624511719
7 409.24615478515625
8 386.7641906738281
9 365.86639404296875
10 346.4895935058594
11 328.38134765625
12 311.321044921875
13 295.1859130859375
14 279.9111022949219
15 265.3537292480469
16 251.47012329101562
17 238.21620178222656
18 225.5154266357422
19 213.4177703857422
20 201.8816375732422
21 190.91513061523438
22 180.43704223632812
23 170.45274353027344
24 160.89804077148438
25 151.8237762451172
26 143.20318603515625
27 135.01651000976562
28 127.2503890991211
29 119.89566802978516
30 112.93623352050781
31 106.3467025756836
32 100.12628936767578
33 94.26721954345703
34 88.75279235839844
35 83.55026245117188
36 78.63842010498047
37 74.00985717773438
38 69.65216827392578
39 65.55877685546875
40 61.69943618774414
41 58.07235336303711
42 54.663333892822266
43 51.4532470703125
44 48.43906784057617
45 45.60804748535156
46 42.948516845703125
47 40.44632339477539
48 38.0987663269043
49 35.89030456542969
50 33.817386627197266
51 31.87490463256836
52 30.045433044433594
53 28.3317928314209
54 26.711170196533203
55 25.188716888427734
56 23.758255004882812
57 22.41607093811035
58 21.153215408325195
59 19.966259002685547
60 18.850297927856445
61 17.800058364868164
62 16.812789916992188
63 15.884719848632812
64 15.010766983032227
65 14.188423156738281
66 13.414408683776855
67 12.687158584594727
68 12.002917289733887
69 11.357102394104004
70 10.749144554138184
71 10.176506042480469
72 9.636682510375977
73 9.127099990844727
74 8.647427558898926
75 8.195331573486328
76 7.76881217956543
77 7.365769863128662
78 6.985809326171875
79 6.6267242431640625
80 6.288330078125
81 5.968692779541016
82 5.667006969451904
83 5.382113933563232
84 5.112699508666992
85 4.857989311218262
86 4.616939067840576
87 4.388705730438232
88 4.173142433166504
89 3.9692182540893555
90 3.7711846828460693
91 3.5839357376098633
92 3.406862497329712
93 3.238633155822754
94 3.079624652862549
95 2.9290859699249268
96 2.786574125289917
97 2.6515753269195557
98 2.523831367492676
99 2.402843713760376
100 2.2881197929382324
101 2.1791563034057617
102 2.0758869647979736
103 1.9780120849609375
104 1.8851284980773926
105 1.7969368696212769
106 1.7132148742675781
107 1.6337207555770874
108 1.5581623315811157
109 1.4865056276321411
110 1.418351411819458
111 1.3534674644470215
112 1.2920397520065308
113 1.2338396310806274
114 1.1785202026367188
115 1.1258915662765503
116 1.0757575035095215
117 1.02800714969635
118 0.9825114011764526
119 0.9392184615135193
120 0.8979032039642334
121 0.8586273193359375
122 0.8211888074874878
123 0.785487711429596
124 0.751417338848114
125 0.7189591526985168
126 0.6880065202713013
127 0.6585051417350769
128 0.6303551197052002
129 0.6034755110740662
130 0.577812910079956
131 0.5533284544944763
132 0.5299572348594666
133 0.5076347589492798
134 0.48630595207214355
135 0.46592482924461365
136 0.4464551508426666
137 0.42786189913749695
138 0.41007301211357117
139 0.3930639922618866
140 0.37679237127304077
141 0.3612179458141327
142 0.3463042080402374
143 0.33205363154411316
144 0.3184202015399933
145 0.305367648601532
146 0.29288652539253235
147 0.2809329628944397
148 0.26950034499168396
149 0.25857698917388916
150 0.24810610711574554
151 0.23807838559150696
152 0.2284877747297287
153 0.21930064260959625
154 0.2104870229959488
155 0.20205038785934448
156 0.19397199153900146
157 0.1862351894378662
158 0.1788187325000763
159 0.17170174419879913
160 0.1648857295513153
161 0.1583477258682251
162 0.15208254754543304
163 0.14608033001422882
164 0.1403254270553589
165 0.1348094642162323
166 0.12951739132404327
167 0.12443705648183823
168 0.11956879496574402
169 0.11489863693714142
170 0.11041805148124695
171 0.10611662268638611
172 0.10198678076267242
173 0.09803025424480438
174 0.09423419088125229
175 0.09058837592601776
176 0.08708396553993225
177 0.08372107148170471
178 0.08049096912145615
179 0.07739023864269257
180 0.07441624253988266
181 0.07156004011631012
182 0.06881721317768097
183 0.06618434190750122
184 0.06365056335926056
185 0.06121790036559105
186 0.05888153612613678
187 0.05663791298866272
188 0.05448247119784355
189 0.05241210758686066
190 0.05042322352528572
191 0.04851217195391655
192 0.04667716845870018
193 0.0449124276638031
194 0.04321586713194847
195 0.04158831387758255
196 0.0400247797369957
197 0.03852325305342674
198 0.037077371031045914
199 0.035688064992427826
200 0.03435344994068146
201 0.03307032957673073
202 0.031836505979299545
203 0.03064887970685959
204 0.02950727753341198
205 0.02840867079794407
206 0.02735305204987526
207 0.02633744478225708
208 0.025360243394970894
209 0.024420922622084618
210 0.023517709225416183
211 0.022649569436907768
212 0.02181362360715866
213 0.021009910851716995
214 0.02023611217737198
215 0.01949133537709713
216 0.018775179982185364
217 0.018085351213812828
218 0.01742120087146759
219 0.016782531514763832
220 0.01616804115474224
221 0.01557731069624424
222 0.0150080481544137
223 0.014459973201155663
224 0.013933264650404453
225 0.013425253331661224
226 0.012936292216181755
227 0.012466149404644966
228 0.01201368123292923
229 0.011578096076846123
230 0.011158942244946957
231 0.01075562834739685
232 0.01036736462265253
233 0.009993433021008968
234 0.009632847271859646
235 0.009285446256399155
236 0.008950967341661453
237 0.008628775365650654
238 0.008318527601659298
239 0.008019879460334778
240 0.007732123602181673
241 0.007454842794686556
242 0.007187744602560997
243 0.006930519361048937
244 0.006682741921395063
245 0.006443711929023266
246 0.006213604938238859
247 0.005992049816995859
248 0.005778325721621513
249 0.005572348367422819
250 0.0053739179857075214
251 0.0051827263087034225
252 0.004998541437089443
253 0.0048210336826741695
254 0.004650148097425699
255 0.0044858697801828384
256 0.004327588714659214
257 0.004174947272986174
258 0.004027785733342171
259 0.003886024234816432
260 0.003749283030629158
261 0.0036174803972244263
262 0.003490536008030176
263 0.003368035424500704
264 0.003250078298151493
265 0.0031361759174615145
266 0.00302642653696239
267 0.0029205356258898973
268 0.002818475244566798
269 0.002720012329518795
270 0.002625041175633669
271 0.0025335412938147783
272 0.002445319900289178
273 0.0023601797875016928
274 0.002278027357533574
275 0.0021988162770867348
276 0.0021225172095000744
277 0.002048766240477562
278 0.001977651845663786
279 0.001909002778120339
280 0.0018428987823426723
281 0.0017790936399251223
282 0.0017175156390294433
283 0.0016581463860347867
284 0.0016008376842364669
285 0.001545572536997497
286 0.001492185634560883
287 0.0014406659174710512
288 0.0013909981353208423
289 0.001343107083812356
290 0.0012968492228537798
291 0.001252254587598145
292 0.0012091916287317872
293 0.0011676442809402943
294 0.0011275571305304766
295 0.0010888638207688928
296 0.0010514851892367005
297 0.0010154461488127708
298 0.000980663695372641
299 0.0009470616932958364
300 0.0009146758820861578
301 0.0008833879255689681
302 0.0008531895000487566
303 0.0008240292663685977
304 0.0007958969799801707
305 0.0007687676115892828
306 0.0007425606017932296
307 0.0007172446930781007
308 0.0006928190123289824
309 0.0006692357710562646
310 0.0006464504403993487
311 0.0006244621472433209
312 0.0006032446399331093
313 0.000582739943638444
314 0.0005629737861454487
315 0.0005438689258880913
316 0.0005254318821243942
317 0.000507628545165062
318 0.0004904278903268278
319 0.0004738371935673058
320 0.00045780802611261606
321 0.0004423396603669971
322 0.0004273819795344025
323 0.00041294810944236815
324 0.0003990152617916465
325 0.00038555581704713404
326 0.00037254285416565835
327 0.0003599944757297635
328 0.0003478841099422425
329 0.0003361758135724813
330 0.0003248630091547966
331 0.00031393126118928194
332 0.00030338685610331595
333 0.0002931967028416693
334 0.00028334782109595835
335 0.0002738440234679729
336 0.00026465911651030183
337 0.00025577700580470264
338 0.00024721238878555596
339 0.0002389363944530487
340 0.00023093429626896977
341 0.0002232064143754542
342 0.0002157432318199426
343 0.0002085260348394513
344 0.00020157199469394982
345 0.00019484185031615198
346 0.00018835172522813082
347 0.00018207401444669813
348 0.00017599230341147631
349 0.0001701369765214622
350 0.00016446308291051537
351 0.0001589892344782129
352 0.00015370050095953047
353 0.00014858557551633567
354 0.00014364530215971172
355 0.00013888083049096167
356 0.00013426222722046077
357 0.00012981203326489776
358 0.00012551058898679912
359 0.00012134979624534026
360 0.0001173350538010709
361 0.00011344751692377031
362 0.00010968926653731614
363 0.00010606009163893759
364 0.00010255110828438774
365 9.916315320879221e-05
366 9.588249668013304e-05
367 9.271786257158965e-05
368 8.965901361079887e-05
369 8.670437819091603e-05
370 8.38430059957318e-05
371 8.1082915130537e-05
372 7.840731996111572e-05
373 7.582772377645597e-05
374 7.333365647355095e-05
375 7.09160667611286e-05
376 6.858799315523356e-05
377 6.633892189711332e-05
378 6.415893585653976e-05
379 6.205315003171563e-05
380 6.0013175243511796e-05
381 5.804771717521362e-05
382 5.614686961052939e-05
383 5.430621968116611e-05
384 5.252317350823432e-05
385 5.08098273712676e-05
386 4.914585952064954e-05
387 4.754201654577628e-05
388 4.5982586016179994e-05
389 4.4480839278548956e-05
390 4.302767410990782e-05
391 4.162411642028019e-05
392 4.026621172670275e-05
393 3.895243935403414e-05
394 3.768223905353807e-05
395 3.645423930720426e-05
396 3.526875298121013e-05
397 3.4119955671485513e-05
398 3.300657044746913e-05
399 3.1937070161802694e-05
400 3.0898932891432196e-05
401 2.989429958688561e-05
402 2.8924971047672443e-05
403 2.7982978281215765e-05
404 2.7075784601038322e-05
405 2.619785300339572e-05
406 2.534903251216747e-05
407 2.4528815629309975e-05
408 2.373317511228379e-05
409 2.296504499099683e-05
410 2.2223433916224167e-05
411 2.1502768504433334e-05
412 2.0810733985854313e-05
413 2.0136116290814243e-05
414 1.94871099665761e-05
415 1.8857561371987686e-05
416 1.8248481865157373e-05
417 1.7659825971350074e-05
418 1.709317984932568e-05
419 1.6542442608624697e-05
420 1.601038638909813e-05
421 1.5495275874854997e-05
422 1.4995751371316146e-05
423 1.4514217582473066e-05
424 1.404927115800092e-05
425 1.3595476048067212e-05
426 1.3160236449039076e-05
427 1.2739095836877823e-05
428 1.2328780940151773e-05
429 1.1933727364521474e-05
430 1.1550309864105657e-05
431 1.1180987712577917e-05
432 1.0821767318702769e-05
433 1.0477945579623338e-05
434 1.014242570818169e-05
435 9.817216778174043e-06
436 9.503984074399341e-06
437 9.200843123835512e-06
438 8.907125447876751e-06
439 8.622317182016559e-06
440 8.346807589987293e-06
441 8.081625310296658e-06
442 7.824042768334039e-06
443 7.574891242256854e-06
444 7.334377642109757e-06
445 7.100344191712793e-06
446 6.874588052596664e-06
447 6.655742254224606e-06
448 6.443796792154899e-06
449 6.239669801288983e-06
450 6.0421775742725e-06
451 5.851015885127708e-06
452 5.663793217536295e-06
453 5.48445950698806e-06
454 5.311528639140306e-06
455 5.143317594047403e-06
456 4.980490302841645e-06
457 4.8230022002826445e-06
458 4.670817361329682e-06
459 4.521835307969013e-06
460 4.379518031782936e-06
461 4.240881935402285e-06
462 4.107502718397882e-06
463 3.978720087616239e-06
464 3.8523171497217845e-06
465 3.7316608541004825e-06
466 3.613576382122119e-06
467 3.499863396427827e-06
468 3.3898170386237325e-06
469 3.283927981101442e-06
470 3.180781504852348e-06
471 3.080567921642796e-06
472 2.983984586535371e-06
473 2.8897300126118353e-06
474 2.7992423383693676e-06
475 2.710875378397759e-06
476 2.6260568120051175e-06
477 2.5443082449783105e-06
478 2.464045110173174e-06
479 2.388350139881368e-06
480 2.3126829091779655e-06
481 2.2399167391995434e-06
482 2.170649622712517e-06
483 2.102123744407436e-06
484 2.0366337594168726e-06
485 1.9730891835934017e-06
486 1.9116062048851745e-06
487 1.8522756590755307e-06
488 1.7944820456250454e-06
489 1.7384787724950002e-06
490 1.6845817754074233e-06
491 1.6323048157573794e-06
492 1.5809254136911477e-06
493 1.5319302519856137e-06
494 1.4848158116365084e-06
495 1.4380899528987356e-06
496 1.3935248261986999e-06
497 1.3498887483365252e-06
498 1.3083595149510074e-06
499 1.267892798750836e-06

FizzBuzz

FizzBuzz是一个简单的小游戏。游戏规则如下：从1开始往上数数，当遇到3的倍数的时候，说fizz，当遇到5的倍数，说buzz，当遇到15的倍数，就说fizzbuzz，其他情况下则正常数数。

我们可以写一个简单的小程序来决定要返回正常数值还是fizz, buzz 或者 fizzbuzz。

# One-hot encode the desired outputs: [number, "fizz", "buzz", "fizzbuzz"]
def fizz_buzz_encode(i):
    if   i % 15 == 0: return 3
    elif i % 5  == 0: return 2
    elif i % 3  == 0: return 1
    else:             return 0

def fizz_buzz_decode(i, prediction):
    return [str(i), "fizz", "buzz", "fizzbuzz"][prediction]

print(fizz_buzz_decode(1, fizz_buzz_encode(1)))
print(fizz_buzz_decode(2, fizz_buzz_encode(2)))
print(fizz_buzz_decode(5, fizz_buzz_encode(5)))
print(fizz_buzz_decode(12, fizz_buzz_encode(12)))
print(fizz_buzz_decode(15, fizz_buzz_encode(15)))

1
2
buzz
fizz
fizzbuzz

我们首先定义模型的输入与输出(训练数据)

import numpy as np
import torch

NUM_DIGITS = 10

# Represent each input by an array of its binary digits.
def binary_encode(i, num_digits):
    return np.array([i >> d & 1 for d in range(num_digits)])

trX = torch.Tensor([binary_encode(i, NUM_DIGITS) for i in range(101, 2 ** NUM_DIGITS)])
trY = torch.LongTensor([fizz_buzz_encode(i) for i in range(101, 2 ** NUM_DIGITS)])

然后我们用PyTorch定义模型

# Define the model
NUM_HIDDEN = 100
model = torch.nn.Sequential(
    torch.nn.Linear(NUM_DIGITS, NUM_HIDDEN),
    torch.nn.ReLU(),
    torch.nn.Linear(NUM_HIDDEN, 4)
)

为了让我们的模型学会FizzBuzz这个游戏，我们需要定义一个损失函数，和一个优化算法。
这个优化算法会不断优化（降低）损失函数，使得模型的在该任务上取得尽可能低的损失值。
损失值低往往表示我们的模型表现好，损失值高表示我们的模型表现差。
由于FizzBuzz游戏本质上是一个分类问题，我们选用Cross Entropyy Loss函数。
优化函数我们选用Stochastic Gradient Descent。

1 2	loss_fn = torch.nn.CrossEntropyLoss() optimizer = torch.optim.SGD(model.parameters(), lr = 0.05)

以下是模型的训练代码

# Start training it
BATCH_SIZE = 128
for epoch in range(10000):
    for start in range(0, len(trX), BATCH_SIZE):
        end = start + BATCH_SIZE
        batchX = trX[start:end]
        batchY = trY[start:end]

        y_pred = model(batchX)
        loss = loss_fn(y_pred, batchY)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    # Find loss on training data
    loss = loss_fn(model(trX), trY).item()
    print('Epoch:', epoch, 'Loss:', loss)

Epoch: 0 Loss: 1.1985573768615723
Epoch: 1 Loss: 1.1627824306488037
Epoch: 2 Loss: 1.153425931930542
Epoch: 3 Loss: 1.1497533321380615
Epoch: 4 Loss: 1.1479018926620483
Epoch: 5 Loss: 1.1467877626419067
Epoch: 6 Loss: 1.1460249423980713
Epoch: 7 Loss: 1.1454517841339111
Epoch: 8 Loss: 1.1449863910675049
Epoch: 9 Loss: 1.1445919275283813
Epoch: 10 Loss: 1.1442433595657349
Epoch: 11 Loss: 1.1439307928085327
Epoch: 12 Loss: 1.1436454057693481
Epoch: 13 Loss: 1.14337956905365
Epoch: 14 Loss: 1.1431307792663574
Epoch: 15 Loss: 1.1428961753845215
Epoch: 16 Loss: 1.142675757408142
Epoch: 17 Loss: 1.1424572467803955
Epoch: 18 Loss: 1.1422516107559204
Epoch: 19 Loss: 1.1420538425445557
Epoch: 20 Loss: 1.1418650150299072
Epoch: 21 Loss: 1.141685962677002
Epoch: 22 Loss: 1.1415083408355713
Epoch: 23 Loss: 1.1413389444351196
Epoch: 24 Loss: 1.1411713361740112
Epoch: 25 Loss: 1.1410109996795654
Epoch: 26 Loss: 1.1408500671386719
Epoch: 27 Loss: 1.1406961679458618
Epoch: 28 Loss: 1.1405448913574219
Epoch: 29 Loss: 1.1403969526290894
Epoch: 30 Loss: 1.1402522325515747
Epoch: 31 Loss: 1.1401114463806152
Epoch: 32 Loss: 1.1399762630462646
Epoch: 33 Loss: 1.1398409605026245
Epoch: 34 Loss: 1.139709711074829
Epoch: 35 Loss: 1.1395775079727173
Epoch: 36 Loss: 1.139445185661316
Epoch: 37 Loss: 1.1393134593963623
Epoch: 38 Loss: 1.1391855478286743
Epoch: 39 Loss: 1.1390624046325684
Epoch: 40 Loss: 1.1389377117156982
Epoch: 41 Loss: 1.1388169527053833
Epoch: 42 Loss: 1.1386950016021729
Epoch: 43 Loss: 1.1385750770568848
Epoch: 44 Loss: 1.1384531259536743
Epoch: 45 Loss: 1.1383368968963623
Epoch: 46 Loss: 1.1382218599319458
Epoch: 47 Loss: 1.1381083726882935
Epoch: 48 Loss: 1.1379945278167725
Epoch: 49 Loss: 1.1378828287124634
Epoch: 50 Loss: 1.1377702951431274
Epoch: 51 Loss: 1.1376607418060303
Epoch: 52 Loss: 1.1375516653060913
Epoch: 53 Loss: 1.1374434232711792
Epoch: 54 Loss: 1.1373358964920044
Epoch: 55 Loss: 1.1372319459915161
Epoch: 56 Loss: 1.1371287107467651
Epoch: 57 Loss: 1.1370246410369873
Epoch: 58 Loss: 1.1369221210479736
Epoch: 59 Loss: 1.1368229389190674
Epoch: 60 Loss: 1.1367197036743164
Epoch: 61 Loss: 1.1366175413131714
Epoch: 62 Loss: 1.1365183591842651
Epoch: 63 Loss: 1.1364164352416992
Epoch: 64 Loss: 1.1363164186477661
Epoch: 65 Loss: 1.1362184286117554
Epoch: 66 Loss: 1.136118769645691
Epoch: 67 Loss: 1.1360193490982056
Epoch: 68 Loss: 1.135920524597168
Epoch: 69 Loss: 1.1358208656311035
Epoch: 70 Loss: 1.1357228755950928
Epoch: 71 Loss: 1.1356228590011597
Epoch: 72 Loss: 1.135527491569519
Epoch: 73 Loss: 1.1354278326034546
Epoch: 74 Loss: 1.1353328227996826
Epoch: 75 Loss: 1.1352288722991943
Epoch: 76 Loss: 1.135129451751709
Epoch: 77 Loss: 1.1350293159484863
Epoch: 78 Loss: 1.1349287033081055
Epoch: 79 Loss: 1.1348257064819336
Epoch: 80 Loss: 1.1347209215164185
Epoch: 81 Loss: 1.1346193552017212
Epoch: 82 Loss: 1.134515643119812
Epoch: 83 Loss: 1.13441002368927
Epoch: 84 Loss: 1.1343063116073608
Epoch: 85 Loss: 1.134202003479004
Epoch: 86 Loss: 1.1340950727462769
Epoch: 87 Loss: 1.1339876651763916
Epoch: 88 Loss: 1.1338807344436646
Epoch: 89 Loss: 1.1337720155715942
Epoch: 90 Loss: 1.1336610317230225
Epoch: 91 Loss: 1.1335523128509521
Epoch: 92 Loss: 1.1334378719329834
Epoch: 93 Loss: 1.1333235502243042
Epoch: 94 Loss: 1.1332066059112549
Epoch: 95 Loss: 1.1330877542495728
Epoch: 96 Loss: 1.1329704523086548
Epoch: 97 Loss: 1.1328511238098145
Epoch: 98 Loss: 1.1327301263809204
Epoch: 99 Loss: 1.1326102018356323
Epoch: 100 Loss: 1.1324888467788696
Epoch: 101 Loss: 1.132361650466919
Epoch: 102 Loss: 1.1322351694107056
Epoch: 103 Loss: 1.1321076154708862
Epoch: 104 Loss: 1.131980061531067
Epoch: 105 Loss: 1.1318477392196655
Epoch: 106 Loss: 1.1317113637924194
Epoch: 107 Loss: 1.131577968597412
Epoch: 108 Loss: 1.131436824798584
Epoch: 109 Loss: 1.131296992301941
Epoch: 110 Loss: 1.1311523914337158
Epoch: 111 Loss: 1.1310100555419922
Epoch: 112 Loss: 1.1308666467666626
Epoch: 113 Loss: 1.1307142972946167
Epoch: 114 Loss: 1.1305700540542603
Epoch: 115 Loss: 1.1304125785827637
Epoch: 116 Loss: 1.1302579641342163
Epoch: 117 Loss: 1.1300987005233765
Epoch: 118 Loss: 1.1299388408660889
Epoch: 119 Loss: 1.1297751665115356
Epoch: 120 Loss: 1.1296147108078003
Epoch: 121 Loss: 1.1294395923614502
Epoch: 122 Loss: 1.1292656660079956
Epoch: 123 Loss: 1.1290934085845947
Epoch: 124 Loss: 1.1289095878601074
Epoch: 125 Loss: 1.1287299394607544
Epoch: 126 Loss: 1.128541111946106
Epoch: 127 Loss: 1.1283516883850098
Epoch: 128 Loss: 1.1281630992889404
Epoch: 129 Loss: 1.1279628276824951
Epoch: 130 Loss: 1.1277685165405273
Epoch: 131 Loss: 1.1275688409805298
Epoch: 132 Loss: 1.1273595094680786
Epoch: 133 Loss: 1.1271624565124512
Epoch: 134 Loss: 1.1269479990005493
Epoch: 135 Loss: 1.126736044883728
Epoch: 136 Loss: 1.1265151500701904
Epoch: 137 Loss: 1.1262929439544678
Epoch: 138 Loss: 1.126067876815796
Epoch: 139 Loss: 1.1258388757705688
Epoch: 140 Loss: 1.1256027221679688
Epoch: 141 Loss: 1.1253615617752075
Epoch: 142 Loss: 1.1251219511032104
Epoch: 143 Loss: 1.1248878240585327
Epoch: 144 Loss: 1.1246533393859863
Epoch: 145 Loss: 1.1244161128997803
Epoch: 146 Loss: 1.124180555343628
Epoch: 147 Loss: 1.1239362955093384
Epoch: 148 Loss: 1.123681902885437
Epoch: 149 Loss: 1.123428463935852
Epoch: 150 Loss: 1.123167872428894
Epoch: 151 Loss: 1.122915267944336
Epoch: 152 Loss: 1.1226478815078735
Epoch: 153 Loss: 1.1223746538162231
Epoch: 154 Loss: 1.1220675706863403
Epoch: 155 Loss: 1.121840476989746
Epoch: 156 Loss: 1.121732234954834
Epoch: 157 Loss: 1.1213226318359375
Epoch: 158 Loss: 1.1212854385375977
Epoch: 159 Loss: 1.1210198402404785
Epoch: 160 Loss: 1.1205651760101318
Epoch: 161 Loss: 1.1203441619873047
Epoch: 162 Loss: 1.1200261116027832
Epoch: 163 Loss: 1.1197220087051392
Epoch: 164 Loss: 1.1194864511489868
Epoch: 165 Loss: 1.11925208568573
Epoch: 166 Loss: 1.11906099319458
Epoch: 167 Loss: 1.1189343929290771
Epoch: 168 Loss: 1.1183561086654663
Epoch: 169 Loss: 1.118335247039795
Epoch: 170 Loss: 1.117862582206726
Epoch: 171 Loss: 1.117429256439209
Epoch: 172 Loss: 1.117123007774353
Epoch: 173 Loss: 1.11692476272583
Epoch: 174 Loss: 1.116502046585083
Epoch: 175 Loss: 1.1162279844284058
Epoch: 176 Loss: 1.1158732175827026
Epoch: 177 Loss: 1.1155357360839844
Epoch: 178 Loss: 1.1152691841125488
Epoch: 179 Loss: 1.1149173974990845
Epoch: 180 Loss: 1.114915132522583
Epoch: 181 Loss: 1.1143337488174438
Epoch: 182 Loss: 1.1142256259918213
Epoch: 183 Loss: 1.1140440702438354
Epoch: 184 Loss: 1.113386869430542
Epoch: 185 Loss: 1.1129437685012817
Epoch: 186 Loss: 1.1130049228668213
Epoch: 187 Loss: 1.112176537513733
Epoch: 188 Loss: 1.112320065498352
Epoch: 189 Loss: 1.1115479469299316
Epoch: 190 Loss: 1.1112663745880127
Epoch: 191 Loss: 1.1113265752792358
Epoch: 192 Loss: 1.1106488704681396
Epoch: 193 Loss: 1.1101278066635132
Epoch: 194 Loss: 1.1101809740066528
Epoch: 195 Loss: 1.10922372341156
Epoch: 196 Loss: 1.1093976497650146
Epoch: 197 Loss: 1.1084682941436768
Epoch: 198 Loss: 1.1081998348236084
Epoch: 199 Loss: 1.1081395149230957
Epoch: 200 Loss: 1.1074204444885254
Epoch: 201 Loss: 1.1070994138717651
Epoch: 202 Loss: 1.1065834760665894
Epoch: 203 Loss: 1.106745719909668
Epoch: 204 Loss: 1.1057730913162231
Epoch: 205 Loss: 1.1053791046142578
Epoch: 206 Loss: 1.1055858135223389
Epoch: 207 Loss: 1.1045254468917847
Epoch: 208 Loss: 1.1041629314422607
Epoch: 209 Loss: 1.1043661832809448
Epoch: 210 Loss: 1.1032700538635254
Epoch: 211 Loss: 1.1030840873718262
Epoch: 212 Loss: 1.1025011539459229
Epoch: 213 Loss: 1.1027953624725342
Epoch: 214 Loss: 1.1017405986785889
Epoch: 215 Loss: 1.1013529300689697
Epoch: 216 Loss: 1.1016074419021606
Epoch: 217 Loss: 1.1008079051971436
Epoch: 218 Loss: 1.1000947952270508
Epoch: 219 Loss: 1.100416898727417
Epoch: 220 Loss: 1.099312424659729
Epoch: 221 Loss: 1.0988670587539673
Epoch: 222 Loss: 1.0990958213806152
Epoch: 223 Loss: 1.0980117321014404
Epoch: 224 Loss: 1.0975043773651123
Epoch: 225 Loss: 1.0979152917861938
Epoch: 226 Loss: 1.0966074466705322
Epoch: 227 Loss: 1.0962252616882324
Epoch: 228 Loss: 1.0957673788070679
Epoch: 229 Loss: 1.095954179763794
Epoch: 230 Loss: 1.0950711965560913
Epoch: 231 Loss: 1.0953128337860107
Epoch: 232 Loss: 1.0946013927459717
Epoch: 233 Loss: 1.093567967414856
Epoch: 234 Loss: 1.0931967496871948
Epoch: 235 Loss: 1.0929981470108032
Epoch: 236 Loss: 1.093286395072937
Epoch: 237 Loss: 1.0922595262527466
Epoch: 238 Loss: 1.0914015769958496
Epoch: 239 Loss: 1.0910273790359497
Epoch: 240 Loss: 1.0907626152038574
Epoch: 241 Loss: 1.0897701978683472
Epoch: 242 Loss: 1.0906466245651245
Epoch: 243 Loss: 1.0892077684402466
Epoch: 244 Loss: 1.08943772315979
Epoch: 245 Loss: 1.0882622003555298
Epoch: 246 Loss: 1.0882519483566284
Epoch: 247 Loss: 1.0873475074768066
Epoch: 248 Loss: 1.0877617597579956
Epoch: 249 Loss: 1.0865530967712402
Epoch: 250 Loss: 1.0860053300857544
Epoch: 251 Loss: 1.086185336112976
Epoch: 252 Loss: 1.0847878456115723
Epoch: 253 Loss: 1.0858310461044312
Epoch: 254 Loss: 1.084405779838562
Epoch: 255 Loss: 1.0837531089782715
Epoch: 256 Loss: 1.082969069480896
Epoch: 257 Loss: 1.0835047960281372
Epoch: 258 Loss: 1.0821843147277832
Epoch: 259 Loss: 1.0821112394332886
Epoch: 260 Loss: 1.0810046195983887
Epoch: 261 Loss: 1.0806779861450195
Epoch: 262 Loss: 1.0805836915969849
Epoch: 263 Loss: 1.080398678779602
Epoch: 264 Loss: 1.0796117782592773
Epoch: 265 Loss: 1.0799351930618286
Epoch: 266 Loss: 1.0785459280014038
Epoch: 267 Loss: 1.0790445804595947
Epoch: 268 Loss: 1.0778568983078003
Epoch: 269 Loss: 1.0781453847885132
Epoch: 270 Loss: 1.07644522190094
Epoch: 271 Loss: 1.0758777856826782
Epoch: 272 Loss: 1.0769966840744019
Epoch: 273 Loss: 1.0758044719696045
Epoch: 274 Loss: 1.0747944116592407
Epoch: 275 Loss: 1.0752646923065186
Epoch: 276 Loss: 1.0738685131072998
Epoch: 277 Loss: 1.0736333131790161
Epoch: 278 Loss: 1.0732215642929077
Epoch: 279 Loss: 1.0725804567337036
Epoch: 280 Loss: 1.0719611644744873
Epoch: 281 Loss: 1.072200894355774
Epoch: 282 Loss: 1.0711501836776733
Epoch: 283 Loss: 1.0706900358200073
Epoch: 284 Loss: 1.069989562034607
Epoch: 285 Loss: 1.0697171688079834
Epoch: 286 Loss: 1.0689090490341187
Epoch: 287 Loss: 1.06852126121521
Epoch: 288 Loss: 1.0678025484085083
Epoch: 289 Loss: 1.0676409006118774
Epoch: 290 Loss: 1.0668482780456543
Epoch: 291 Loss: 1.0668519735336304
Epoch: 292 Loss: 1.0660663843154907
Epoch: 293 Loss: 1.0655699968338013
Epoch: 294 Loss: 1.0647932291030884
Epoch: 295 Loss: 1.064287543296814
Epoch: 296 Loss: 1.0637518167495728
Epoch: 297 Loss: 1.0637747049331665
Epoch: 298 Loss: 1.0626537799835205
Epoch: 299 Loss: 1.0621955394744873
Epoch: 300 Loss: 1.061999797821045
Epoch: 301 Loss: 1.0616999864578247
Epoch: 302 Loss: 1.0610554218292236
Epoch: 303 Loss: 1.0604358911514282
Epoch: 304 Loss: 1.0598989725112915
Epoch: 305 Loss: 1.0594773292541504
Epoch: 306 Loss: 1.0588374137878418
Epoch: 307 Loss: 1.0580638647079468
Epoch: 308 Loss: 1.0574591159820557
Epoch: 309 Loss: 1.0578587055206299
Epoch: 310 Loss: 1.0566651821136475
Epoch: 311 Loss: 1.0568552017211914
Epoch: 312 Loss: 1.055396556854248
Epoch: 313 Loss: 1.055452823638916
Epoch: 314 Loss: 1.0545830726623535
Epoch: 315 Loss: 1.0538941621780396
Epoch: 316 Loss: 1.05329430103302
Epoch: 317 Loss: 1.0525377988815308
Epoch: 318 Loss: 1.0526529550552368
Epoch: 319 Loss: 1.0531365871429443
Epoch: 320 Loss: 1.0510408878326416
Epoch: 321 Loss: 1.0514711141586304
Epoch: 322 Loss: 1.0492733716964722
Epoch: 323 Loss: 1.0501182079315186
Epoch: 324 Loss: 1.0490694046020508
Epoch: 325 Loss: 1.048269510269165
Epoch: 326 Loss: 1.0478101968765259
Epoch: 327 Loss: 1.0470324754714966
Epoch: 328 Loss: 1.0475497245788574
Epoch: 329 Loss: 1.0455683469772339
Epoch: 330 Loss: 1.0453155040740967
Epoch: 331 Loss: 1.0459259748458862
Epoch: 332 Loss: 1.043872594833374
Epoch: 333 Loss: 1.0438199043273926
Epoch: 334 Loss: 1.0426747798919678
Epoch: 335 Loss: 1.0422818660736084
Epoch: 336 Loss: 1.041212558746338
Epoch: 337 Loss: 1.0425331592559814
Epoch: 338 Loss: 1.0403465032577515
Epoch: 339 Loss: 1.0394083261489868
Epoch: 340 Loss: 1.039574384689331
Epoch: 341 Loss: 1.0382622480392456
Epoch: 342 Loss: 1.0376906394958496
Epoch: 343 Loss: 1.0368410348892212
Epoch: 344 Loss: 1.036706566810608
Epoch: 345 Loss: 1.0353500843048096
Epoch: 346 Loss: 1.0348291397094727
Epoch: 347 Loss: 1.0358717441558838
Epoch: 348 Loss: 1.0335254669189453
Epoch: 349 Loss: 1.0345847606658936
Epoch: 350 Loss: 1.032273530960083
Epoch: 351 Loss: 1.0314823389053345
Epoch: 352 Loss: 1.031983494758606
Epoch: 353 Loss: 1.029603362083435
Epoch: 354 Loss: 1.028960943222046
Epoch: 355 Loss: 1.0292367935180664
Epoch: 356 Loss: 1.028082013130188
Epoch: 357 Loss: 1.027571439743042
Epoch: 358 Loss: 1.0259982347488403
Epoch: 359 Loss: 1.0260169506072998
Epoch: 360 Loss: 1.0260775089263916
Epoch: 361 Loss: 1.0243809223175049
Epoch: 362 Loss: 1.0236375331878662
Epoch: 363 Loss: 1.0237226486206055
Epoch: 364 Loss: 1.0223020315170288
Epoch: 365 Loss: 1.0213770866394043
Epoch: 366 Loss: 1.0207571983337402
Epoch: 367 Loss: 1.019388198852539
Epoch: 368 Loss: 1.021054744720459
Epoch: 369 Loss: 1.01825749874115
Epoch: 370 Loss: 1.018271803855896
Epoch: 371 Loss: 1.0170068740844727
Epoch: 372 Loss: 1.016271710395813
Epoch: 373 Loss: 1.0151135921478271
Epoch: 374 Loss: 1.0154625177383423
Epoch: 375 Loss: 1.0134416818618774
Epoch: 376 Loss: 1.014142394065857
Epoch: 377 Loss: 1.0134960412979126
Epoch: 378 Loss: 1.0116677284240723
Epoch: 379 Loss: 1.0112065076828003
Epoch: 380 Loss: 1.0083858966827393
Epoch: 381 Loss: 1.0087339878082275
Epoch: 382 Loss: 1.0070652961730957
Epoch: 383 Loss: 1.0077122449874878
Epoch: 384 Loss: 1.0102046728134155
Epoch: 385 Loss: 1.005751132965088
Epoch: 386 Loss: 1.006459355354309
Epoch: 387 Loss: 1.004715919494629
Epoch: 388 Loss: 1.0042649507522583
Epoch: 389 Loss: 1.0034807920455933
Epoch: 390 Loss: 1.0022400617599487
Epoch: 391 Loss: 1.0024681091308594
Epoch: 392 Loss: 0.9997951984405518
Epoch: 393 Loss: 0.9993601441383362
Epoch: 394 Loss: 0.9984546303749084
Epoch: 395 Loss: 0.9988020062446594
Epoch: 396 Loss: 0.9962315559387207
Epoch: 397 Loss: 0.9950620532035828
Epoch: 398 Loss: 0.9969995021820068
Epoch: 399 Loss: 0.9945296049118042
Epoch: 400 Loss: 0.9937195181846619
Epoch: 401 Loss: 0.9952678084373474
Epoch: 402 Loss: 0.9923511743545532
Epoch: 403 Loss: 0.9920569658279419
Epoch: 404 Loss: 0.9915186166763306
Epoch: 405 Loss: 0.9892837405204773
Epoch: 406 Loss: 0.9914341568946838
Epoch: 407 Loss: 0.9887666702270508
Epoch: 408 Loss: 0.9879904389381409
Epoch: 409 Loss: 0.9856729507446289
Epoch: 410 Loss: 0.9878743290901184
Epoch: 411 Loss: 0.9855794906616211
Epoch: 412 Loss: 0.9843447208404541
Epoch: 413 Loss: 0.983269214630127
Epoch: 414 Loss: 0.9818732142448425
Epoch: 415 Loss: 0.9817400574684143
Epoch: 416 Loss: 0.9813429713249207
Epoch: 417 Loss: 0.9789907336235046
Epoch: 418 Loss: 0.979008138179779
Epoch: 419 Loss: 0.9795718789100647
Epoch: 420 Loss: 0.9774047136306763
Epoch: 421 Loss: 0.9765068888664246
Epoch: 422 Loss: 0.9748741388320923
Epoch: 423 Loss: 0.9742738008499146
Epoch: 424 Loss: 0.9754096269607544
Epoch: 425 Loss: 0.9723442196846008
Epoch: 426 Loss: 0.9715316891670227
Epoch: 427 Loss: 0.9697376489639282
Epoch: 428 Loss: 0.9712840914726257
Epoch: 429 Loss: 0.9705616235733032
Epoch: 430 Loss: 0.9673716425895691
Epoch: 431 Loss: 0.9661984443664551
Epoch: 432 Loss: 0.9674557447433472
Epoch: 433 Loss: 0.9676986336708069
Epoch: 434 Loss: 0.9633025527000427
Epoch: 435 Loss: 0.9643653035163879
Epoch: 436 Loss: 0.964128315448761
Epoch: 437 Loss: 0.9616101980209351
Epoch: 438 Loss: 0.9634836912155151
Epoch: 439 Loss: 0.9604705572128296
Epoch: 440 Loss: 0.9581515789031982
Epoch: 441 Loss: 0.9577563405036926
Epoch: 442 Loss: 0.9562257528305054
Epoch: 443 Loss: 0.9571819305419922
Epoch: 444 Loss: 0.9551123976707458
Epoch: 445 Loss: 0.9575338363647461
Epoch: 446 Loss: 0.9533542394638062
Epoch: 447 Loss: 0.9512361288070679
Epoch: 448 Loss: 0.9541177749633789
Epoch: 449 Loss: 0.9504536986351013
Epoch: 450 Loss: 0.9507186412811279
Epoch: 451 Loss: 0.9497722387313843
Epoch: 452 Loss: 0.9479610323905945
Epoch: 453 Loss: 0.948036789894104
Epoch: 454 Loss: 0.9449342489242554
Epoch: 455 Loss: 0.9470672011375427
Epoch: 456 Loss: 0.9451227784156799
Epoch: 457 Loss: 0.94453364610672
Epoch: 458 Loss: 0.9419916272163391
Epoch: 459 Loss: 0.942990243434906
Epoch: 460 Loss: 0.9411762356758118
Epoch: 461 Loss: 0.9390718936920166
Epoch: 462 Loss: 0.9401019811630249
Epoch: 463 Loss: 0.9369909763336182
Epoch: 464 Loss: 0.9381493926048279
Epoch: 465 Loss: 0.9372242093086243
Epoch: 466 Loss: 0.9353963136672974
Epoch: 467 Loss: 0.9343509674072266
Epoch: 468 Loss: 0.9333584308624268
Epoch: 469 Loss: 0.9320292472839355
Epoch: 470 Loss: 0.9314770698547363
Epoch: 471 Loss: 0.9295525550842285
Epoch: 472 Loss: 0.9277319312095642
Epoch: 473 Loss: 0.9274896383285522
Epoch: 474 Loss: 0.92833411693573
Epoch: 475 Loss: 0.9258905649185181
Epoch: 476 Loss: 0.9267155528068542
Epoch: 477 Loss: 0.9257214665412903
Epoch: 478 Loss: 0.9260300993919373
Epoch: 479 Loss: 0.9229422807693481
Epoch: 480 Loss: 0.9214460849761963
Epoch: 481 Loss: 0.9198169708251953
Epoch: 482 Loss: 0.9207642078399658
Epoch: 483 Loss: 0.91725754737854
Epoch: 484 Loss: 0.9164453148841858
Epoch: 485 Loss: 0.9144068360328674
Epoch: 486 Loss: 0.917241632938385
Epoch: 487 Loss: 0.9145542979240417
Epoch: 488 Loss: 0.9170951247215271
Epoch: 489 Loss: 0.9129472970962524
Epoch: 490 Loss: 0.9113500118255615
Epoch: 491 Loss: 0.9109109044075012
Epoch: 492 Loss: 0.9116742610931396
Epoch: 493 Loss: 0.9092674851417542
Epoch: 494 Loss: 0.907427191734314
Epoch: 495 Loss: 0.9072278738021851
Epoch: 496 Loss: 0.9100430607795715
Epoch: 497 Loss: 0.906478762626648
Epoch: 498 Loss: 0.905219316482544
Epoch: 499 Loss: 0.9046626091003418
Epoch: 500 Loss: 0.9022524356842041
Epoch: 501 Loss: 0.8999878168106079
Epoch: 502 Loss: 0.8999902009963989
Epoch: 503 Loss: 0.9018612504005432
Epoch: 504 Loss: 0.8977088928222656
Epoch: 505 Loss: 0.8961044549942017
Epoch: 506 Loss: 0.8953272700309753
Epoch: 507 Loss: 0.8978896141052246
Epoch: 508 Loss: 0.8943119049072266
Epoch: 509 Loss: 0.8952851891517639
Epoch: 510 Loss: 0.8921042680740356
Epoch: 511 Loss: 0.8915343284606934
Epoch: 512 Loss: 0.8894192576408386
Epoch: 513 Loss: 0.8894717693328857
Epoch: 514 Loss: 0.8872820138931274
Epoch: 515 Loss: 0.8908304572105408
Epoch: 516 Loss: 0.8855993747711182
Epoch: 517 Loss: 0.8842307329177856
Epoch: 518 Loss: 0.8853597044944763
Epoch: 519 Loss: 0.8822887539863586
Epoch: 520 Loss: 0.8833261132240295
Epoch: 521 Loss: 0.8827701210975647
Epoch: 522 Loss: 0.8802623748779297
Epoch: 523 Loss: 0.8804686069488525
Epoch: 524 Loss: 0.8795179128646851
Epoch: 525 Loss: 0.8775599002838135
Epoch: 526 Loss: 0.8767753839492798
Epoch: 527 Loss: 0.8764278292655945
Epoch: 528 Loss: 0.8742743134498596
Epoch: 529 Loss: 0.8745076060295105
Epoch: 530 Loss: 0.8740688562393188
Epoch: 531 Loss: 0.8744255304336548
Epoch: 532 Loss: 0.872100293636322
Epoch: 533 Loss: 0.8696321249008179
Epoch: 534 Loss: 0.8697847723960876
Epoch: 535 Loss: 0.8682160973548889
Epoch: 536 Loss: 0.8678727149963379
Epoch: 537 Loss: 0.8652617931365967
Epoch: 538 Loss: 0.8663988709449768
Epoch: 539 Loss: 0.8650947213172913
Epoch: 540 Loss: 0.8644912242889404
Epoch: 541 Loss: 0.8626392483711243
Epoch: 542 Loss: 0.8613178133964539
Epoch: 543 Loss: 0.8604719042778015
Epoch: 544 Loss: 0.85905921459198
Epoch: 545 Loss: 0.8582139015197754
Epoch: 546 Loss: 0.8573085069656372
Epoch: 547 Loss: 0.85661381483078
Epoch: 548 Loss: 0.8549367189407349
Epoch: 549 Loss: 0.8546679019927979
Epoch: 550 Loss: 0.8531329035758972
Epoch: 551 Loss: 0.8529951572418213
Epoch: 552 Loss: 0.8515518307685852
Epoch: 553 Loss: 0.8511015772819519
Epoch: 554 Loss: 0.8496919274330139
Epoch: 555 Loss: 0.8493216633796692
Epoch: 556 Loss: 0.8482795357704163
Epoch: 557 Loss: 0.8472979664802551
Epoch: 558 Loss: 0.8463637828826904
Epoch: 559 Loss: 0.8458774089813232
Epoch: 560 Loss: 0.8443583250045776
Epoch: 561 Loss: 0.843368411064148
Epoch: 562 Loss: 0.842404305934906
Epoch: 563 Loss: 0.841273307800293
Epoch: 564 Loss: 0.8401919007301331
Epoch: 565 Loss: 0.8395553827285767
Epoch: 566 Loss: 0.8386046886444092
Epoch: 567 Loss: 0.8374596238136292
Epoch: 568 Loss: 0.836303174495697
Epoch: 569 Loss: 0.8357030749320984
Epoch: 570 Loss: 0.8343329429626465
Epoch: 571 Loss: 0.8336198925971985
Epoch: 572 Loss: 0.8325223922729492
Epoch: 573 Loss: 0.8317338228225708
Epoch: 574 Loss: 0.830647885799408
Epoch: 575 Loss: 0.8296640515327454
Epoch: 576 Loss: 0.8288283348083496
Epoch: 577 Loss: 0.8277338147163391
Epoch: 578 Loss: 0.8272828459739685
Epoch: 579 Loss: 0.8256038427352905
Epoch: 580 Loss: 0.8246623873710632
Epoch: 581 Loss: 0.8236204981803894
Epoch: 582 Loss: 0.8230394721031189
Epoch: 583 Loss: 0.8215901851654053
Epoch: 584 Loss: 0.8205523490905762
Epoch: 585 Loss: 0.8193093538284302
Epoch: 586 Loss: 0.8185722231864929
Epoch: 587 Loss: 0.8176712393760681
Epoch: 588 Loss: 0.8163898587226868
Epoch: 589 Loss: 0.8159493803977966
Epoch: 590 Loss: 0.814298152923584
Epoch: 591 Loss: 0.8130376935005188
Epoch: 592 Loss: 0.8124143481254578
Epoch: 593 Loss: 0.8106762170791626
Epoch: 594 Loss: 0.8102102875709534
Epoch: 595 Loss: 0.8094911575317383
Epoch: 596 Loss: 0.8082305192947388
Epoch: 597 Loss: 0.8069621920585632
Epoch: 598 Loss: 0.8062826991081238
Epoch: 599 Loss: 0.8053361773490906
Epoch: 600 Loss: 0.8042335510253906
Epoch: 601 Loss: 0.8029847741127014
Epoch: 602 Loss: 0.8024702072143555
Epoch: 603 Loss: 0.8010755777359009
Epoch: 604 Loss: 0.8003895282745361
Epoch: 605 Loss: 0.7991005182266235
Epoch: 606 Loss: 0.7985773086547852
Epoch: 607 Loss: 0.7973751425743103
Epoch: 608 Loss: 0.7969022989273071
Epoch: 609 Loss: 0.7957760095596313
Epoch: 610 Loss: 0.7943004369735718
Epoch: 611 Loss: 0.7937070727348328
Epoch: 612 Loss: 0.7921230792999268
Epoch: 613 Loss: 0.7915046215057373
Epoch: 614 Loss: 0.7903255820274353
Epoch: 615 Loss: 0.789604127407074
Epoch: 616 Loss: 0.7882571220397949
Epoch: 617 Loss: 0.787748396396637
Epoch: 618 Loss: 0.7862234711647034
Epoch: 619 Loss: 0.7856805324554443
Epoch: 620 Loss: 0.7841756939888
Epoch: 621 Loss: 0.7841836810112
Epoch: 622 Loss: 0.7823336720466614
Epoch: 623 Loss: 0.7813976407051086
Epoch: 624 Loss: 0.7807126045227051
Epoch: 625 Loss: 0.779725968837738
Epoch: 626 Loss: 0.77848219871521
Epoch: 627 Loss: 0.7776318192481995
Epoch: 628 Loss: 0.7765378355979919
Epoch: 629 Loss: 0.7761077284812927
Epoch: 630 Loss: 0.7757213711738586
Epoch: 631 Loss: 0.7734090685844421
Epoch: 632 Loss: 0.7730903625488281
Epoch: 633 Loss: 0.7719579935073853
Epoch: 634 Loss: 0.7707761526107788
Epoch: 635 Loss: 0.7697094678878784
Epoch: 636 Loss: 0.7686758041381836
Epoch: 637 Loss: 0.7677622437477112
Epoch: 638 Loss: 0.7663392424583435
Epoch: 639 Loss: 0.7656968832015991
Epoch: 640 Loss: 0.7649017572402954
Epoch: 641 Loss: 0.7636702656745911
Epoch: 642 Loss: 0.7628093361854553
Epoch: 643 Loss: 0.7620269656181335
Epoch: 644 Loss: 0.7605606913566589
Epoch: 645 Loss: 0.7599747180938721
Epoch: 646 Loss: 0.7588849663734436
Epoch: 647 Loss: 0.7579389810562134
Epoch: 648 Loss: 0.7568257451057434
Epoch: 649 Loss: 0.7558368444442749
Epoch: 650 Loss: 0.754461944103241
Epoch: 651 Loss: 0.7547708749771118
Epoch: 652 Loss: 0.7533630728721619
Epoch: 653 Loss: 0.751636266708374
Epoch: 654 Loss: 0.7516142129898071
Epoch: 655 Loss: 0.7495936751365662
Epoch: 656 Loss: 0.7489213347434998
Epoch: 657 Loss: 0.7477341890335083
Epoch: 658 Loss: 0.7469910979270935
Epoch: 659 Loss: 0.745832085609436
Epoch: 660 Loss: 0.7448437809944153
Epoch: 661 Loss: 0.7439432740211487
Epoch: 662 Loss: 0.7426746487617493
Epoch: 663 Loss: 0.7436144948005676
Epoch: 664 Loss: 0.7410452961921692
Epoch: 665 Loss: 0.7395506501197815
Epoch: 666 Loss: 0.7388194799423218
Epoch: 667 Loss: 0.7372956275939941
Epoch: 668 Loss: 0.7363263368606567
Epoch: 669 Loss: 0.7361353635787964
Epoch: 670 Loss: 0.7340744137763977
Epoch: 671 Loss: 0.7337360382080078
Epoch: 672 Loss: 0.7324140667915344
Epoch: 673 Loss: 0.7309783697128296
Epoch: 674 Loss: 0.7295973896980286
Epoch: 675 Loss: 0.7287357449531555
Epoch: 676 Loss: 0.7277989387512207
Epoch: 677 Loss: 0.7270971536636353
Epoch: 678 Loss: 0.7255963087081909
Epoch: 679 Loss: 0.7247186303138733
Epoch: 680 Loss: 0.7262864708900452
Epoch: 681 Loss: 0.7228329181671143
Epoch: 682 Loss: 0.7218132615089417
Epoch: 683 Loss: 0.7211008667945862
Epoch: 684 Loss: 0.7198137044906616
Epoch: 685 Loss: 0.7184186577796936
Epoch: 686 Loss: 0.7178533673286438
Epoch: 687 Loss: 0.7166286110877991
Epoch: 688 Loss: 0.7158548831939697
Epoch: 689 Loss: 0.7143851518630981
Epoch: 690 Loss: 0.7136035561561584
Epoch: 691 Loss: 0.7120166420936584
Epoch: 692 Loss: 0.7125294804573059
Epoch: 693 Loss: 0.7103173732757568
Epoch: 694 Loss: 0.7094016075134277
Epoch: 695 Loss: 0.7087060213088989
Epoch: 696 Loss: 0.7069821953773499
Epoch: 697 Loss: 0.7064146399497986
Epoch: 698 Loss: 0.7051834464073181
Epoch: 699 Loss: 0.7047383189201355
Epoch: 700 Loss: 0.706749439239502
Epoch: 701 Loss: 0.7017005681991577
Epoch: 702 Loss: 0.7010787725448608
Epoch: 703 Loss: 0.6999382376670837
Epoch: 704 Loss: 0.6990237236022949
Epoch: 705 Loss: 0.6996415257453918
Epoch: 706 Loss: 0.6967211961746216
Epoch: 707 Loss: 0.6958404779434204
Epoch: 708 Loss: 0.696262538433075
Epoch: 709 Loss: 0.6936417818069458
Epoch: 710 Loss: 0.6931089758872986
Epoch: 711 Loss: 0.6987453699111938
Epoch: 712 Loss: 0.6905298829078674
Epoch: 713 Loss: 0.6894145607948303
Epoch: 714 Loss: 0.6952964067459106
Epoch: 715 Loss: 0.6869481205940247
Epoch: 716 Loss: 0.6872397065162659
Epoch: 717 Loss: 0.6920738816261292
Epoch: 718 Loss: 0.6840365529060364
Epoch: 719 Loss: 0.6845012903213501
Epoch: 720 Loss: 0.6825422644615173
Epoch: 721 Loss: 0.681422233581543
Epoch: 722 Loss: 0.6810960173606873
Epoch: 723 Loss: 0.68485426902771
Epoch: 724 Loss: 0.6807733178138733
Epoch: 725 Loss: 0.6777227520942688
Epoch: 726 Loss: 0.6786050796508789
Epoch: 727 Loss: 0.6754059195518494
Epoch: 728 Loss: 0.6751314401626587
Epoch: 729 Loss: 0.674085795879364
Epoch: 730 Loss: 0.6726557016372681
Epoch: 731 Loss: 0.6714085936546326
Epoch: 732 Loss: 0.677187442779541
Epoch: 733 Loss: 0.6704249978065491
Epoch: 734 Loss: 0.6710174679756165
Epoch: 735 Loss: 0.6682985424995422
Epoch: 736 Loss: 0.670428454875946
Epoch: 737 Loss: 0.6646300554275513
Epoch: 738 Loss: 0.6659622192382812
Epoch: 739 Loss: 0.6641559600830078
Epoch: 740 Loss: 0.663042426109314
Epoch: 741 Loss: 0.6645637154579163
Epoch: 742 Loss: 0.6625307202339172
Epoch: 743 Loss: 0.6620370149612427
Epoch: 744 Loss: 0.6588280200958252
Epoch: 745 Loss: 0.6598482728004456
Epoch: 746 Loss: 0.6566712260246277
Epoch: 747 Loss: 0.65584796667099
Epoch: 748 Loss: 0.6541467905044556
Epoch: 749 Loss: 0.6550030708312988
Epoch: 750 Loss: 0.6521152257919312
Epoch: 751 Loss: 0.6533358097076416
Epoch: 752 Loss: 0.6501272916793823
Epoch: 753 Loss: 0.6518148183822632
Epoch: 754 Loss: 0.6470169425010681
Epoch: 755 Loss: 0.6496713757514954
Epoch: 756 Loss: 0.6446613669395447
Epoch: 757 Loss: 0.6441842317581177
Epoch: 758 Loss: 0.6448515057563782
Epoch: 759 Loss: 0.6426706314086914
Epoch: 760 Loss: 0.6419839262962341
Epoch: 761 Loss: 0.6426022052764893
Epoch: 762 Loss: 0.6384557485580444
Epoch: 763 Loss: 0.6397757530212402
Epoch: 764 Loss: 0.6365482211112976
Epoch: 765 Loss: 0.6391438841819763
Epoch: 766 Loss: 0.6346348524093628
Epoch: 767 Loss: 0.632697582244873
Epoch: 768 Loss: 0.6333869695663452
Epoch: 769 Loss: 0.6308565139770508
Epoch: 770 Loss: 0.6325890421867371
Epoch: 771 Loss: 0.630275547504425
Epoch: 772 Loss: 0.6305156350135803
Epoch: 773 Loss: 0.6267229318618774
Epoch: 774 Loss: 0.6290708780288696
Epoch: 775 Loss: 0.6243835687637329
Epoch: 776 Loss: 0.6240254044532776
Epoch: 777 Loss: 0.625835657119751
Epoch: 778 Loss: 0.6209756731987
Epoch: 779 Loss: 0.6201242208480835
Epoch: 780 Loss: 0.6201583743095398
Epoch: 781 Loss: 0.6178663969039917
Epoch: 782 Loss: 0.6186249256134033
Epoch: 783 Loss: 0.6161662340164185
Epoch: 784 Loss: 0.6161298751831055
Epoch: 785 Loss: 0.6136629581451416
Epoch: 786 Loss: 0.6136850118637085
Epoch: 787 Loss: 0.61044842004776
Epoch: 788 Loss: 0.6117837429046631
Epoch: 789 Loss: 0.6085878014564514
Epoch: 790 Loss: 0.6085320711135864
Epoch: 791 Loss: 0.6071110963821411
Epoch: 792 Loss: 0.6051099896430969
Epoch: 793 Loss: 0.6061027646064758
Epoch: 794 Loss: 0.6024988293647766
Epoch: 795 Loss: 0.6017165780067444
Epoch: 796 Loss: 0.6026225686073303
Epoch: 797 Loss: 0.6023247241973877
Epoch: 798 Loss: 0.5984103083610535
Epoch: 799 Loss: 0.5973348021507263
Epoch: 800 Loss: 0.5980710387229919
Epoch: 801 Loss: 0.5957676768302917
Epoch: 802 Loss: 0.5959773063659668
Epoch: 803 Loss: 0.5933066606521606
Epoch: 804 Loss: 0.5934427976608276
Epoch: 805 Loss: 0.5910834074020386
Epoch: 806 Loss: 0.5919994711875916
Epoch: 807 Loss: 0.5903356671333313
Epoch: 808 Loss: 0.5896205306053162
Epoch: 809 Loss: 0.5871273875236511
Epoch: 810 Loss: 0.5881789326667786
Epoch: 811 Loss: 0.5845828056335449
Epoch: 812 Loss: 0.5844382643699646
Epoch: 813 Loss: 0.5830240249633789
Epoch: 814 Loss: 0.581424355506897
Epoch: 815 Loss: 0.582291305065155
Epoch: 816 Loss: 0.5797979831695557
Epoch: 817 Loss: 0.5798642039299011
Epoch: 818 Loss: 0.5776983499526978
Epoch: 819 Loss: 0.5782819390296936
Epoch: 820 Loss: 0.5770540833473206
Epoch: 821 Loss: 0.5747533440589905
Epoch: 822 Loss: 0.5743867754936218
Epoch: 823 Loss: 0.57242751121521
Epoch: 824 Loss: 0.5714877247810364
Epoch: 825 Loss: 0.5716946125030518
Epoch: 826 Loss: 0.5697606801986694
Epoch: 827 Loss: 0.5691030621528625
Epoch: 828 Loss: 0.5671324729919434
Epoch: 829 Loss: 0.567787230014801
Epoch: 830 Loss: 0.5653936862945557
Epoch: 831 Loss: 0.5653029084205627
Epoch: 832 Loss: 0.5637446045875549
Epoch: 833 Loss: 0.5637123584747314
Epoch: 834 Loss: 0.5615712404251099
Epoch: 835 Loss: 0.5609508156776428
Epoch: 836 Loss: 0.5592301487922668
Epoch: 837 Loss: 0.5593835711479187
Epoch: 838 Loss: 0.5571972727775574
Epoch: 839 Loss: 0.5571238994598389
Epoch: 840 Loss: 0.5551512241363525
Epoch: 841 Loss: 0.5562667846679688
Epoch: 842 Loss: 0.5541607737541199
Epoch: 843 Loss: 0.5528791546821594
Epoch: 844 Loss: 0.5523701310157776
Epoch: 845 Loss: 0.5507773160934448
Epoch: 846 Loss: 0.5511012673377991
Epoch: 847 Loss: 0.5484548211097717
Epoch: 848 Loss: 0.5481991767883301
Epoch: 849 Loss: 0.54603511095047
Epoch: 850 Loss: 0.5449923872947693
Epoch: 851 Loss: 0.5465194582939148
Epoch: 852 Loss: 0.5438029170036316
Epoch: 853 Loss: 0.5439284443855286
Epoch: 854 Loss: 0.5414514541625977
Epoch: 855 Loss: 0.5426289439201355
Epoch: 856 Loss: 0.5395417809486389
Epoch: 857 Loss: 0.5391501784324646
Epoch: 858 Loss: 0.538345456123352
Epoch: 859 Loss: 0.53753262758255
Epoch: 860 Loss: 0.5366492867469788
Epoch: 861 Loss: 0.5349016785621643
Epoch: 862 Loss: 0.5356824994087219
Epoch: 863 Loss: 0.5331833958625793
Epoch: 864 Loss: 0.5321553945541382
Epoch: 865 Loss: 0.5311114192008972
Epoch: 866 Loss: 0.5315322875976562
Epoch: 867 Loss: 0.5288553237915039
Epoch: 868 Loss: 0.5279883146286011
Epoch: 869 Loss: 0.5279349088668823
Epoch: 870 Loss: 0.5262319445610046
Epoch: 871 Loss: 0.5265958905220032
Epoch: 872 Loss: 0.524341344833374
Epoch: 873 Loss: 0.5247524380683899
Epoch: 874 Loss: 0.5227304100990295
Epoch: 875 Loss: 0.521461009979248
Epoch: 876 Loss: 0.521816074848175
Epoch: 877 Loss: 0.519683301448822
Epoch: 878 Loss: 0.5187017321586609
Epoch: 879 Loss: 0.517973780632019
Epoch: 880 Loss: 0.5166322588920593
Epoch: 881 Loss: 0.5156897902488708
Epoch: 882 Loss: 0.5149350166320801
Epoch: 883 Loss: 0.515521228313446
Epoch: 884 Loss: 0.5131540298461914
Epoch: 885 Loss: 0.5130569338798523
Epoch: 886 Loss: 0.5112212300300598
Epoch: 887 Loss: 0.5103504061698914
Epoch: 888 Loss: 0.5099576711654663
Epoch: 889 Loss: 0.5083349347114563
Epoch: 890 Loss: 0.5086829662322998
Epoch: 891 Loss: 0.5063547492027283
Epoch: 892 Loss: 0.5057884454727173
Epoch: 893 Loss: 0.5047110915184021
Epoch: 894 Loss: 0.5044532418251038
Epoch: 895 Loss: 0.5028647184371948
Epoch: 896 Loss: 0.5025187134742737
Epoch: 897 Loss: 0.5011277794837952
Epoch: 898 Loss: 0.5013135671615601
Epoch: 899 Loss: 0.49957528710365295
Epoch: 900 Loss: 0.4986625909805298
Epoch: 901 Loss: 0.49741482734680176
Epoch: 902 Loss: 0.4977731704711914
Epoch: 903 Loss: 0.4953472316265106
Epoch: 904 Loss: 0.4952242374420166
Epoch: 905 Loss: 0.4938902258872986
Epoch: 906 Loss: 0.49317502975463867
Epoch: 907 Loss: 0.4919264018535614
Epoch: 908 Loss: 0.4910534620285034
Epoch: 909 Loss: 0.4900832176208496
Epoch: 910 Loss: 0.48924699425697327
Epoch: 911 Loss: 0.4882091283798218
Epoch: 912 Loss: 0.48800942301750183
Epoch: 913 Loss: 0.48642101883888245
Epoch: 914 Loss: 0.48593780398368835
Epoch: 915 Loss: 0.4847544729709625
Epoch: 916 Loss: 0.4840855598449707
Epoch: 917 Loss: 0.48283952474594116
Epoch: 918 Loss: 0.4824126958847046
Epoch: 919 Loss: 0.4814419150352478
Epoch: 920 Loss: 0.48033109307289124
Epoch: 921 Loss: 0.47957363724708557
Epoch: 922 Loss: 0.4788576662540436
Epoch: 923 Loss: 0.47774600982666016
Epoch: 924 Loss: 0.4766923785209656
Epoch: 925 Loss: 0.47632038593292236
Epoch: 926 Loss: 0.47496142983436584
Epoch: 927 Loss: 0.47406062483787537
Epoch: 928 Loss: 0.4733272194862366
Epoch: 929 Loss: 0.47265470027923584
Epoch: 930 Loss: 0.47169414162635803
Epoch: 931 Loss: 0.47101226449012756
Epoch: 932 Loss: 0.46968236565589905
Epoch: 933 Loss: 0.4689907133579254
Epoch: 934 Loss: 0.4679068922996521
Epoch: 935 Loss: 0.46710965037345886
Epoch: 936 Loss: 0.46612557768821716
Epoch: 937 Loss: 0.4653874337673187
Epoch: 938 Loss: 0.46460434794425964
Epoch: 939 Loss: 0.46362143754959106
Epoch: 940 Loss: 0.4629553556442261
Epoch: 941 Loss: 0.46182510256767273
Epoch: 942 Loss: 0.4610803723335266
Epoch: 943 Loss: 0.46015045046806335
Epoch: 944 Loss: 0.45944467186927795
Epoch: 945 Loss: 0.45854833722114563
Epoch: 946 Loss: 0.45733413100242615
Epoch: 947 Loss: 0.4566118121147156
Epoch: 948 Loss: 0.4560736417770386
Epoch: 949 Loss: 0.4551783800125122
Epoch: 950 Loss: 0.4545534551143646
Epoch: 951 Loss: 0.45311403274536133
Epoch: 952 Loss: 0.4522720277309418
Epoch: 953 Loss: 0.45133981108665466
Epoch: 954 Loss: 0.45052093267440796
Epoch: 955 Loss: 0.4496237635612488
Epoch: 956 Loss: 0.44904059171676636
Epoch: 957 Loss: 0.44812753796577454
Epoch: 958 Loss: 0.4470473825931549
Epoch: 959 Loss: 0.44627243280410767
Epoch: 960 Loss: 0.4455432593822479
Epoch: 961 Loss: 0.4445818066596985
Epoch: 962 Loss: 0.44390788674354553
Epoch: 963 Loss: 0.4428580105304718
Epoch: 964 Loss: 0.44209739565849304
Epoch: 965 Loss: 0.4412752091884613
Epoch: 966 Loss: 0.4405112564563751
Epoch: 967 Loss: 0.43954744935035706
Epoch: 968 Loss: 0.4389835596084595
Epoch: 969 Loss: 0.43788567185401917
Epoch: 970 Loss: 0.4372676908969879
Epoch: 971 Loss: 0.4362795054912567
Epoch: 972 Loss: 0.43587997555732727
Epoch: 973 Loss: 0.43463775515556335
Epoch: 974 Loss: 0.4341322183609009
Epoch: 975 Loss: 0.43314242362976074
Epoch: 976 Loss: 0.43277522921562195
Epoch: 977 Loss: 0.4314977526664734
Epoch: 978 Loss: 0.431158185005188
Epoch: 979 Loss: 0.4300278425216675
Epoch: 980 Loss: 0.4294261634349823
Epoch: 981 Loss: 0.428317129611969
Epoch: 982 Loss: 0.4279724955558777
Epoch: 983 Loss: 0.42682239413261414
Epoch: 984 Loss: 0.4261079728603363
Epoch: 985 Loss: 0.4252490699291229
Epoch: 986 Loss: 0.42444321513175964
Epoch: 987 Loss: 0.4235529899597168
Epoch: 988 Loss: 0.4230058491230011
Epoch: 989 Loss: 0.42231252789497375
Epoch: 990 Loss: 0.42139932513237
Epoch: 991 Loss: 0.42051318287849426
Epoch: 992 Loss: 0.41989409923553467
Epoch: 993 Loss: 0.418841153383255
Epoch: 994 Loss: 0.4183008670806885
Epoch: 995 Loss: 0.41741514205932617
Epoch: 996 Loss: 0.4166196286678314
Epoch: 997 Loss: 0.41601917147636414
Epoch: 998 Loss: 0.41565075516700745
Epoch: 999 Loss: 0.41466429829597473
Epoch: 1000 Loss: 0.4137318432331085
Epoch: 1001 Loss: 0.41294288635253906
Epoch: 1002 Loss: 0.4119333028793335
Epoch: 1003 Loss: 0.41133445501327515
Epoch: 1004 Loss: 0.4105459451675415
Epoch: 1005 Loss: 0.40974539518356323
Epoch: 1006 Loss: 0.40888872742652893
Epoch: 1007 Loss: 0.4081083834171295
Epoch: 1008 Loss: 0.4073196351528168
Epoch: 1009 Loss: 0.40658214688301086
Epoch: 1010 Loss: 0.4063185155391693
Epoch: 1011 Loss: 0.4049651622772217
Epoch: 1012 Loss: 0.4043104350566864
Epoch: 1013 Loss: 0.40402883291244507
Epoch: 1014 Loss: 0.4029316306114197
Epoch: 1015 Loss: 0.4023118317127228
Epoch: 1016 Loss: 0.40174177289009094
Epoch: 1017 Loss: 0.40070852637290955
Epoch: 1018 Loss: 0.3999391496181488
Epoch: 1019 Loss: 0.39911943674087524
Epoch: 1020 Loss: 0.39844387769699097
Epoch: 1021 Loss: 0.39751383662223816
Epoch: 1022 Loss: 0.3968171775341034
Epoch: 1023 Loss: 0.3961794972419739
Epoch: 1024 Loss: 0.3957788050174713
Epoch: 1025 Loss: 0.3947219252586365
Epoch: 1026 Loss: 0.39402130246162415
Epoch: 1027 Loss: 0.3933540880680084
Epoch: 1028 Loss: 0.3927133083343506
Epoch: 1029 Loss: 0.39170774817466736
Epoch: 1030 Loss: 0.3910624384880066
Epoch: 1031 Loss: 0.3903416097164154
Epoch: 1032 Loss: 0.3894534409046173
Epoch: 1033 Loss: 0.38910678029060364
Epoch: 1034 Loss: 0.38804876804351807
Epoch: 1035 Loss: 0.38740020990371704
Epoch: 1036 Loss: 0.3867513835430145
Epoch: 1037 Loss: 0.3857179582118988
Epoch: 1038 Loss: 0.38509324193000793
Epoch: 1039 Loss: 0.3845541775226593
Epoch: 1040 Loss: 0.38379499316215515
Epoch: 1041 Loss: 0.3831019997596741
Epoch: 1042 Loss: 0.3822353482246399
Epoch: 1043 Loss: 0.38198646903038025
Epoch: 1044 Loss: 0.38093075156211853
Epoch: 1045 Loss: 0.380243718624115
Epoch: 1046 Loss: 0.3793095350265503
Epoch: 1047 Loss: 0.37865519523620605
Epoch: 1048 Loss: 0.378172904253006
Epoch: 1049 Loss: 0.3773215711116791
Epoch: 1050 Loss: 0.3766757845878601
Epoch: 1051 Loss: 0.3759629428386688
Epoch: 1052 Loss: 0.3750542998313904
Epoch: 1053 Loss: 0.374594509601593
Epoch: 1054 Loss: 0.37376558780670166
Epoch: 1055 Loss: 0.3735867440700531
Epoch: 1056 Loss: 0.3726474940776825
Epoch: 1057 Loss: 0.371855229139328
Epoch: 1058 Loss: 0.3708328604698181
Epoch: 1059 Loss: 0.37032079696655273
Epoch: 1060 Loss: 0.36958491802215576
Epoch: 1061 Loss: 0.36897796392440796
Epoch: 1062 Loss: 0.3681003749370575
Epoch: 1063 Loss: 0.3675887882709503
Epoch: 1064 Loss: 0.36701077222824097
Epoch: 1065 Loss: 0.36618825793266296
Epoch: 1066 Loss: 0.3652178645133972
Epoch: 1067 Loss: 0.36470189690589905
Epoch: 1068 Loss: 0.3640306293964386
Epoch: 1069 Loss: 0.36320096254348755
Epoch: 1070 Loss: 0.36279797554016113
Epoch: 1071 Loss: 0.3618871867656708
Epoch: 1072 Loss: 0.3613240718841553
Epoch: 1073 Loss: 0.360715389251709
Epoch: 1074 Loss: 0.36002469062805176
Epoch: 1075 Loss: 0.35904765129089355
Epoch: 1076 Loss: 0.3584807515144348
Epoch: 1077 Loss: 0.35777556896209717
Epoch: 1078 Loss: 0.3570973575115204
Epoch: 1079 Loss: 0.3563465476036072
Epoch: 1080 Loss: 0.35574519634246826
Epoch: 1081 Loss: 0.355184406042099
Epoch: 1082 Loss: 0.35461658239364624
Epoch: 1083 Loss: 0.35367414355278015
Epoch: 1084 Loss: 0.35292306542396545
Epoch: 1085 Loss: 0.3525625765323639
Epoch: 1086 Loss: 0.35179445147514343
Epoch: 1087 Loss: 0.35109588503837585
Epoch: 1088 Loss: 0.3506033420562744
Epoch: 1089 Loss: 0.34968337416648865
Epoch: 1090 Loss: 0.34913405776023865
Epoch: 1091 Loss: 0.3486005365848541
Epoch: 1092 Loss: 0.34773585200309753
Epoch: 1093 Loss: 0.34722617268562317
Epoch: 1094 Loss: 0.3465431332588196
Epoch: 1095 Loss: 0.345831036567688
Epoch: 1096 Loss: 0.3452007472515106
Epoch: 1097 Loss: 0.34455713629722595
Epoch: 1098 Loss: 0.3438655138015747
Epoch: 1099 Loss: 0.3432844579219818
Epoch: 1100 Loss: 0.3426769971847534
Epoch: 1101 Loss: 0.34220004081726074
Epoch: 1102 Loss: 0.34157854318618774
Epoch: 1103 Loss: 0.3407893776893616
Epoch: 1104 Loss: 0.3401757478713989
Epoch: 1105 Loss: 0.3393135666847229
Epoch: 1106 Loss: 0.33879223465919495
Epoch: 1107 Loss: 0.33812612295150757
Epoch: 1108 Loss: 0.337352991104126
Epoch: 1109 Loss: 0.3367348611354828
Epoch: 1110 Loss: 0.3364359736442566
Epoch: 1111 Loss: 0.3356296718120575
Epoch: 1112 Loss: 0.3351328372955322
Epoch: 1113 Loss: 0.3347057104110718
Epoch: 1114 Loss: 0.33375540375709534
Epoch: 1115 Loss: 0.33327966928482056
Epoch: 1116 Loss: 0.3325839042663574
Epoch: 1117 Loss: 0.33193889260292053
Epoch: 1118 Loss: 0.33116793632507324
Epoch: 1119 Loss: 0.33062461018562317
Epoch: 1120 Loss: 0.33008116483688354
Epoch: 1121 Loss: 0.32947665452957153
Epoch: 1122 Loss: 0.32872864603996277
Epoch: 1123 Loss: 0.3282651901245117
Epoch: 1124 Loss: 0.3275107443332672
Epoch: 1125 Loss: 0.3270872235298157
Epoch: 1126 Loss: 0.3263515532016754
Epoch: 1127 Loss: 0.3256600797176361
Epoch: 1128 Loss: 0.3252382278442383
Epoch: 1129 Loss: 0.32451575994491577
Epoch: 1130 Loss: 0.32399317622184753
Epoch: 1131 Loss: 0.3233492374420166
Epoch: 1132 Loss: 0.3226448893547058
Epoch: 1133 Loss: 0.32222265005111694
Epoch: 1134 Loss: 0.3216770589351654
Epoch: 1135 Loss: 0.3209243416786194
Epoch: 1136 Loss: 0.32048842310905457
Epoch: 1137 Loss: 0.3196019232273102
Epoch: 1138 Loss: 0.3194229304790497
Epoch: 1139 Loss: 0.3185880482196808
Epoch: 1140 Loss: 0.3180570900440216
Epoch: 1141 Loss: 0.31729236245155334
Epoch: 1142 Loss: 0.31699851155281067
Epoch: 1143 Loss: 0.31631654500961304
Epoch: 1144 Loss: 0.3158511519432068
Epoch: 1145 Loss: 0.31524065136909485
Epoch: 1146 Loss: 0.31449997425079346
Epoch: 1147 Loss: 0.3138991594314575
Epoch: 1148 Loss: 0.3132588267326355
Epoch: 1149 Loss: 0.3127800524234772
Epoch: 1150 Loss: 0.3125785291194916
Epoch: 1151 Loss: 0.3117516040802002
Epoch: 1152 Loss: 0.31117331981658936
Epoch: 1153 Loss: 0.31073591113090515
Epoch: 1154 Loss: 0.31006893515586853
Epoch: 1155 Loss: 0.30954959988594055
Epoch: 1156 Loss: 0.30891865491867065
Epoch: 1157 Loss: 0.30839866399765015
Epoch: 1158 Loss: 0.30766138434410095
Epoch: 1159 Loss: 0.307339608669281
Epoch: 1160 Loss: 0.3066447377204895
Epoch: 1161 Loss: 0.3063533306121826
Epoch: 1162 Loss: 0.3057190179824829
Epoch: 1163 Loss: 0.3050052523612976
Epoch: 1164 Loss: 0.3045117259025574
Epoch: 1165 Loss: 0.3039288818836212
Epoch: 1166 Loss: 0.3033660650253296
Epoch: 1167 Loss: 0.30292296409606934
Epoch: 1168 Loss: 0.30246996879577637
Epoch: 1169 Loss: 0.3018036186695099
Epoch: 1170 Loss: 0.30124518275260925
Epoch: 1171 Loss: 0.30081889033317566
Epoch: 1172 Loss: 0.30033525824546814
Epoch: 1173 Loss: 0.29980161786079407
Epoch: 1174 Loss: 0.29912006855010986
Epoch: 1175 Loss: 0.2988583743572235
Epoch: 1176 Loss: 0.2981162965297699
Epoch: 1177 Loss: 0.29770079255104065
Epoch: 1178 Loss: 0.2975713610649109
Epoch: 1179 Loss: 0.29679325222969055
Epoch: 1180 Loss: 0.2962171137332916
Epoch: 1181 Loss: 0.2957089841365814
Epoch: 1182 Loss: 0.29490482807159424
Epoch: 1183 Loss: 0.29451945424079895
Epoch: 1184 Loss: 0.29434797167778015
Epoch: 1185 Loss: 0.2934170961380005
Epoch: 1186 Loss: 0.29313111305236816
Epoch: 1187 Loss: 0.2925609350204468
Epoch: 1188 Loss: 0.2921147346496582
Epoch: 1189 Loss: 0.2915200889110565
Epoch: 1190 Loss: 0.2909773290157318
Epoch: 1191 Loss: 0.29054829478263855
Epoch: 1192 Loss: 0.2901393473148346
Epoch: 1193 Loss: 0.28978589177131653
Epoch: 1194 Loss: 0.28917062282562256
Epoch: 1195 Loss: 0.2883111238479614
Epoch: 1196 Loss: 0.28792300820350647
Epoch: 1197 Loss: 0.28759899735450745
Epoch: 1198 Loss: 0.2869868874549866
Epoch: 1199 Loss: 0.2865521311759949
Epoch: 1200 Loss: 0.2860497534275055
Epoch: 1201 Loss: 0.28571921586990356
Epoch: 1202 Loss: 0.2850522994995117
Epoch: 1203 Loss: 0.2844160795211792
Epoch: 1204 Loss: 0.28397685289382935
Epoch: 1205 Loss: 0.2834942042827606
Epoch: 1206 Loss: 0.28318873047828674
Epoch: 1207 Loss: 0.28270313143730164
Epoch: 1208 Loss: 0.2819063365459442
Epoch: 1209 Loss: 0.28178441524505615
Epoch: 1210 Loss: 0.28128695487976074
Epoch: 1211 Loss: 0.280429869890213
Epoch: 1212 Loss: 0.2801244556903839
Epoch: 1213 Loss: 0.2796393036842346
Epoch: 1214 Loss: 0.2791718542575836
Epoch: 1215 Loss: 0.27889955043792725
Epoch: 1216 Loss: 0.27813151478767395
Epoch: 1217 Loss: 0.27764227986335754
Epoch: 1218 Loss: 0.27702978253364563
Epoch: 1219 Loss: 0.2765936255455017
Epoch: 1220 Loss: 0.27628445625305176
Epoch: 1221 Loss: 0.2758007049560547
Epoch: 1222 Loss: 0.27542534470558167
Epoch: 1223 Loss: 0.2747534513473511
Epoch: 1224 Loss: 0.2742491066455841
Epoch: 1225 Loss: 0.2739832103252411
Epoch: 1226 Loss: 0.2732876241207123
Epoch: 1227 Loss: 0.2730964720249176
Epoch: 1228 Loss: 0.2727319896221161
Epoch: 1229 Loss: 0.27227988839149475
Epoch: 1230 Loss: 0.271914005279541
Epoch: 1231 Loss: 0.2713056802749634
Epoch: 1232 Loss: 0.27077075839042664
Epoch: 1233 Loss: 0.2701992094516754
Epoch: 1234 Loss: 0.2697504758834839
Epoch: 1235 Loss: 0.26925793290138245
Epoch: 1236 Loss: 0.2689872682094574
Epoch: 1237 Loss: 0.2683658003807068
Epoch: 1238 Loss: 0.26786598563194275
Epoch: 1239 Loss: 0.2674296796321869
Epoch: 1240 Loss: 0.2670622169971466
Epoch: 1241 Loss: 0.2668967843055725
Epoch: 1242 Loss: 0.2662796676158905
Epoch: 1243 Loss: 0.26583343744277954
Epoch: 1244 Loss: 0.2653738558292389
Epoch: 1245 Loss: 0.26480498909950256
Epoch: 1246 Loss: 0.26442065834999084
Epoch: 1247 Loss: 0.2641977369785309
Epoch: 1248 Loss: 0.26368701457977295
Epoch: 1249 Loss: 0.2631950080394745
Epoch: 1250 Loss: 0.26296266913414
Epoch: 1251 Loss: 0.2624060809612274
Epoch: 1252 Loss: 0.26190048456192017
Epoch: 1253 Loss: 0.26147016882896423
Epoch: 1254 Loss: 0.26118776202201843
Epoch: 1255 Loss: 0.26067665219306946
Epoch: 1256 Loss: 0.260214626789093
Epoch: 1257 Loss: 0.2596901059150696
Epoch: 1258 Loss: 0.25947943329811096
Epoch: 1259 Loss: 0.25882643461227417
Epoch: 1260 Loss: 0.25851503014564514
Epoch: 1261 Loss: 0.25811558961868286
Epoch: 1262 Loss: 0.25762510299682617
Epoch: 1263 Loss: 0.25752630829811096
Epoch: 1264 Loss: 0.25704267621040344
Epoch: 1265 Loss: 0.2563774585723877
Epoch: 1266 Loss: 0.2559394836425781
Epoch: 1267 Loss: 0.25563713908195496
Epoch: 1268 Loss: 0.2552395462989807
Epoch: 1269 Loss: 0.25482338666915894
Epoch: 1270 Loss: 0.2545318901538849
Epoch: 1271 Loss: 0.25396856665611267
Epoch: 1272 Loss: 0.2535995841026306
Epoch: 1273 Loss: 0.25319042801856995
Epoch: 1274 Loss: 0.2527627646923065
Epoch: 1275 Loss: 0.2523566484451294
Epoch: 1276 Loss: 0.2520396411418915
Epoch: 1277 Loss: 0.2515976130962372
Epoch: 1278 Loss: 0.2511201798915863
Epoch: 1279 Loss: 0.25075966119766235
Epoch: 1280 Loss: 0.2503626346588135
Epoch: 1281 Loss: 0.2499864101409912
Epoch: 1282 Loss: 0.24959996342658997
Epoch: 1283 Loss: 0.24918632209300995
Epoch: 1284 Loss: 0.24875031411647797
Epoch: 1285 Loss: 0.2486734241247177
Epoch: 1286 Loss: 0.2483886480331421
Epoch: 1287 Loss: 0.247640460729599
Epoch: 1288 Loss: 0.24737407267093658
Epoch: 1289 Loss: 0.24686722457408905
Epoch: 1290 Loss: 0.24644288420677185
Epoch: 1291 Loss: 0.2461613416671753
Epoch: 1292 Loss: 0.24559873342514038
Epoch: 1293 Loss: 0.24528048932552338
Epoch: 1294 Loss: 0.24498721957206726
Epoch: 1295 Loss: 0.24464750289916992
Epoch: 1296 Loss: 0.24420972168445587
Epoch: 1297 Loss: 0.243879497051239
Epoch: 1298 Loss: 0.243353009223938
Epoch: 1299 Loss: 0.24299003183841705
Epoch: 1300 Loss: 0.24286028742790222
Epoch: 1301 Loss: 0.24247802793979645
Epoch: 1302 Loss: 0.2421358823776245
Epoch: 1303 Loss: 0.24153383076190948
Epoch: 1304 Loss: 0.24120475351810455
Epoch: 1305 Loss: 0.24100813269615173
Epoch: 1306 Loss: 0.24042655527591705
Epoch: 1307 Loss: 0.2400541454553604
Epoch: 1308 Loss: 0.2397562712430954
Epoch: 1309 Loss: 0.23942463099956512
Epoch: 1310 Loss: 0.23928384482860565
Epoch: 1311 Loss: 0.2385450005531311
Epoch: 1312 Loss: 0.23829130828380585
Epoch: 1313 Loss: 0.23789720237255096
Epoch: 1314 Loss: 0.23779292404651642
Epoch: 1315 Loss: 0.23730409145355225
Epoch: 1316 Loss: 0.23678085207939148
Epoch: 1317 Loss: 0.23647218942642212
Epoch: 1318 Loss: 0.23602284491062164
Epoch: 1319 Loss: 0.23569601774215698
Epoch: 1320 Loss: 0.23525141179561615
Epoch: 1321 Loss: 0.2349250614643097
Epoch: 1322 Loss: 0.23449568450450897
Epoch: 1323 Loss: 0.2345777451992035
Epoch: 1324 Loss: 0.23416222631931305
Epoch: 1325 Loss: 0.23355048894882202
Epoch: 1326 Loss: 0.23322995007038116
Epoch: 1327 Loss: 0.23283782601356506
Epoch: 1328 Loss: 0.23253050446510315
Epoch: 1329 Loss: 0.23210416734218597
Epoch: 1330 Loss: 0.23180176317691803
Epoch: 1331 Loss: 0.23141805827617645
Epoch: 1332 Loss: 0.23119500279426575
Epoch: 1333 Loss: 0.23077908158302307
Epoch: 1334 Loss: 0.23030684888362885
Epoch: 1335 Loss: 0.23010708391666412
Epoch: 1336 Loss: 0.22976671159267426
Epoch: 1337 Loss: 0.22933153808116913
Epoch: 1338 Loss: 0.22897523641586304
Epoch: 1339 Loss: 0.22869619727134705
Epoch: 1340 Loss: 0.22842802107334137
Epoch: 1341 Loss: 0.22830316424369812
Epoch: 1342 Loss: 0.22788682579994202
Epoch: 1343 Loss: 0.2272893488407135
Epoch: 1344 Loss: 0.22722621262073517
Epoch: 1345 Loss: 0.22663111984729767
Epoch: 1346 Loss: 0.2262491136789322
Epoch: 1347 Loss: 0.22596187889575958
Epoch: 1348 Loss: 0.22560568153858185
Epoch: 1349 Loss: 0.2252582311630249
Epoch: 1350 Loss: 0.22487512230873108
Epoch: 1351 Loss: 0.22458085417747498
Epoch: 1352 Loss: 0.22447998821735382
Epoch: 1353 Loss: 0.2240002602338791
Epoch: 1354 Loss: 0.22356551885604858
Epoch: 1355 Loss: 0.2234889715909958
Epoch: 1356 Loss: 0.223052516579628
Epoch: 1357 Loss: 0.22257278859615326
Epoch: 1358 Loss: 0.22238700091838837
Epoch: 1359 Loss: 0.22196230292320251
Epoch: 1360 Loss: 0.221641406416893
Epoch: 1361 Loss: 0.22161716222763062
Epoch: 1362 Loss: 0.2209445983171463
Epoch: 1363 Loss: 0.2206478863954544
Epoch: 1364 Loss: 0.22032888233661652
Epoch: 1365 Loss: 0.219917431473732
Epoch: 1366 Loss: 0.2199840396642685
Epoch: 1367 Loss: 0.21953995525836945
Epoch: 1368 Loss: 0.21906402707099915
Epoch: 1369 Loss: 0.21892724931240082
Epoch: 1370 Loss: 0.2185245305299759
Epoch: 1371 Loss: 0.2180560827255249
Epoch: 1372 Loss: 0.21817925572395325
Epoch: 1373 Loss: 0.21783952414989471
Epoch: 1374 Loss: 0.21741579473018646
Epoch: 1375 Loss: 0.21688516438007355
Epoch: 1376 Loss: 0.21660302579402924
Epoch: 1377 Loss: 0.21643024682998657
Epoch: 1378 Loss: 0.21601806581020355
Epoch: 1379 Loss: 0.21596971154212952
Epoch: 1380 Loss: 0.21545866131782532
Epoch: 1381 Loss: 0.21537533402442932
Epoch: 1382 Loss: 0.21475717425346375
Epoch: 1383 Loss: 0.21438884735107422
Epoch: 1384 Loss: 0.21412204205989838
Epoch: 1385 Loss: 0.21377059817314148
Epoch: 1386 Loss: 0.2134363055229187
Epoch: 1387 Loss: 0.21357853710651398
Epoch: 1388 Loss: 0.21290180087089539
Epoch: 1389 Loss: 0.21281597018241882
Epoch: 1390 Loss: 0.21230600774288177
Epoch: 1391 Loss: 0.2120925486087799
Epoch: 1392 Loss: 0.2117575854063034
Epoch: 1393 Loss: 0.21150128543376923
Epoch: 1394 Loss: 0.21139658987522125
Epoch: 1395 Loss: 0.21110771596431732
Epoch: 1396 Loss: 0.21072624623775482
Epoch: 1397 Loss: 0.21027876436710358
Epoch: 1398 Loss: 0.20998448133468628
Epoch: 1399 Loss: 0.20967832207679749
Epoch: 1400 Loss: 0.2093939185142517
Epoch: 1401 Loss: 0.2093273252248764
Epoch: 1402 Loss: 0.20899318158626556
Epoch: 1403 Loss: 0.2086038440465927
Epoch: 1404 Loss: 0.20826418697834015
Epoch: 1405 Loss: 0.20791789889335632
Epoch: 1406 Loss: 0.20797868072986603
Epoch: 1407 Loss: 0.2076280564069748
Epoch: 1408 Loss: 0.2071039080619812
Epoch: 1409 Loss: 0.20677191019058228
Epoch: 1410 Loss: 0.20673538744449615
Epoch: 1411 Loss: 0.20631514489650726
Epoch: 1412 Loss: 0.2060493677854538
Epoch: 1413 Loss: 0.2056865692138672
Epoch: 1414 Loss: 0.20557957887649536
Epoch: 1415 Loss: 0.2052898108959198
Epoch: 1416 Loss: 0.2048460692167282
Epoch: 1417 Loss: 0.20452582836151123
Epoch: 1418 Loss: 0.20425841212272644
Epoch: 1419 Loss: 0.20400944352149963
Epoch: 1420 Loss: 0.20395518839359283
Epoch: 1421 Loss: 0.2035394161939621
Epoch: 1422 Loss: 0.2034623920917511
Epoch: 1423 Loss: 0.2029036432504654
Epoch: 1424 Loss: 0.20258285105228424
Epoch: 1425 Loss: 0.20238783955574036
Epoch: 1426 Loss: 0.2020205408334732
Epoch: 1427 Loss: 0.20211370289325714
Epoch: 1428 Loss: 0.20154805481433868
Epoch: 1429 Loss: 0.2015133649110794
Epoch: 1430 Loss: 0.20104123651981354
Epoch: 1431 Loss: 0.2008308470249176
Epoch: 1432 Loss: 0.20053894817829132
Epoch: 1433 Loss: 0.20029500126838684
Epoch: 1434 Loss: 0.19991253316402435
Epoch: 1435 Loss: 0.19966930150985718
Epoch: 1436 Loss: 0.19938553869724274
Epoch: 1437 Loss: 0.19927692413330078
Epoch: 1438 Loss: 0.19890519976615906
Epoch: 1439 Loss: 0.1985926330089569
Epoch: 1440 Loss: 0.19825881719589233
Epoch: 1441 Loss: 0.19842156767845154
Epoch: 1442 Loss: 0.19805343449115753
Epoch: 1443 Loss: 0.1976834088563919
Epoch: 1444 Loss: 0.19730299711227417
Epoch: 1445 Loss: 0.19701600074768066
Epoch: 1446 Loss: 0.19706590473651886
Epoch: 1447 Loss: 0.19660593569278717
Epoch: 1448 Loss: 0.19637958705425262
Epoch: 1449 Loss: 0.19620904326438904
Epoch: 1450 Loss: 0.1957329660654068
Epoch: 1451 Loss: 0.19562487304210663
Epoch: 1452 Loss: 0.19522607326507568
Epoch: 1453 Loss: 0.19493722915649414
Epoch: 1454 Loss: 0.19500276446342468
Epoch: 1455 Loss: 0.19447366893291473
Epoch: 1456 Loss: 0.19434276223182678
Epoch: 1457 Loss: 0.19418393075466156
Epoch: 1458 Loss: 0.19366207718849182
Epoch: 1459 Loss: 0.19356633722782135
Epoch: 1460 Loss: 0.19320231676101685
Epoch: 1461 Loss: 0.19321513175964355
Epoch: 1462 Loss: 0.19270752370357513
Epoch: 1463 Loss: 0.19258171319961548
Epoch: 1464 Loss: 0.19222451746463776
Epoch: 1465 Loss: 0.1919592022895813
Epoch: 1466 Loss: 0.1916380077600479
Epoch: 1467 Loss: 0.191635861992836
Epoch: 1468 Loss: 0.19129280745983124
Epoch: 1469 Loss: 0.19123907387256622
Epoch: 1470 Loss: 0.19069494307041168
Epoch: 1471 Loss: 0.19052518904209137
Epoch: 1472 Loss: 0.19025586545467377
Epoch: 1473 Loss: 0.18990284204483032
Epoch: 1474 Loss: 0.18966396152973175
Epoch: 1475 Loss: 0.189625084400177
Epoch: 1476 Loss: 0.1893175095319748
Epoch: 1477 Loss: 0.18901324272155762
Epoch: 1478 Loss: 0.18895335495471954
Epoch: 1479 Loss: 0.18852843344211578
Epoch: 1480 Loss: 0.18848437070846558
Epoch: 1481 Loss: 0.18799670040607452
Epoch: 1482 Loss: 0.18779779970645905
Epoch: 1483 Loss: 0.1876165121793747
Epoch: 1484 Loss: 0.18721625208854675
Epoch: 1485 Loss: 0.18707728385925293
Epoch: 1486 Loss: 0.18685083091259003
Epoch: 1487 Loss: 0.18656812608242035
Epoch: 1488 Loss: 0.18642845749855042
Epoch: 1489 Loss: 0.1862272024154663
Epoch: 1490 Loss: 0.18579180538654327
Epoch: 1491 Loss: 0.18566496670246124
Epoch: 1492 Loss: 0.18555355072021484
Epoch: 1493 Loss: 0.1851319819688797
Epoch: 1494 Loss: 0.18510721623897552
Epoch: 1495 Loss: 0.1846039593219757
Epoch: 1496 Loss: 0.1846092790365219
Epoch: 1497 Loss: 0.18419149518013
Epoch: 1498 Loss: 0.18395774066448212
Epoch: 1499 Loss: 0.18397092819213867
Epoch: 1500 Loss: 0.18358492851257324
Epoch: 1501 Loss: 0.1832953244447708
Epoch: 1502 Loss: 0.18296103179454803
Epoch: 1503 Loss: 0.1829986423254013
Epoch: 1504 Loss: 0.18256965279579163
Epoch: 1505 Loss: 0.18248781561851501
Epoch: 1506 Loss: 0.18213754892349243
Epoch: 1507 Loss: 0.18200130760669708
Epoch: 1508 Loss: 0.18185032904148102
Epoch: 1509 Loss: 0.1814313530921936
Epoch: 1510 Loss: 0.18126815557479858
Epoch: 1511 Loss: 0.18105094134807587
Epoch: 1512 Loss: 0.18082892894744873
Epoch: 1513 Loss: 0.18061985075473785
Epoch: 1514 Loss: 0.18032248318195343
Epoch: 1515 Loss: 0.18029481172561646
Epoch: 1516 Loss: 0.18011216819286346
Epoch: 1517 Loss: 0.17963694036006927
Epoch: 1518 Loss: 0.17964716255664825
Epoch: 1519 Loss: 0.1794218122959137
Epoch: 1520 Loss: 0.17909301817417145
Epoch: 1521 Loss: 0.17881590127944946
Epoch: 1522 Loss: 0.1785333752632141
Epoch: 1523 Loss: 0.17843255400657654
Epoch: 1524 Loss: 0.17813564836978912
Epoch: 1525 Loss: 0.1778896152973175
Epoch: 1526 Loss: 0.17781883478164673
Epoch: 1527 Loss: 0.17767056822776794
Epoch: 1528 Loss: 0.1773587465286255
Epoch: 1529 Loss: 0.17716749012470245
Epoch: 1530 Loss: 0.17690008878707886
Epoch: 1531 Loss: 0.17660140991210938
Epoch: 1532 Loss: 0.1763855516910553
Epoch: 1533 Loss: 0.17625881731510162
Epoch: 1534 Loss: 0.17593644559383392
Epoch: 1535 Loss: 0.1758764088153839
Epoch: 1536 Loss: 0.17574337124824524
Epoch: 1537 Loss: 0.175413578748703
Epoch: 1538 Loss: 0.17518416047096252
Epoch: 1539 Loss: 0.17500920593738556
Epoch: 1540 Loss: 0.17467758059501648
Epoch: 1541 Loss: 0.17459087073802948
Epoch: 1542 Loss: 0.174322709441185
Epoch: 1543 Loss: 0.17401807010173798
Epoch: 1544 Loss: 0.17392665147781372
Epoch: 1545 Loss: 0.1738814115524292
Epoch: 1546 Loss: 0.17359468340873718
Epoch: 1547 Loss: 0.17328156530857086
Epoch: 1548 Loss: 0.1730712652206421
Epoch: 1549 Loss: 0.17295733094215393
Epoch: 1550 Loss: 0.17269137501716614
Epoch: 1551 Loss: 0.17263203859329224
Epoch: 1552 Loss: 0.172257661819458
Epoch: 1553 Loss: 0.1721264123916626
Epoch: 1554 Loss: 0.17182648181915283
Epoch: 1555 Loss: 0.17162998020648956
Epoch: 1556 Loss: 0.17140895128250122
Epoch: 1557 Loss: 0.17124900221824646
Epoch: 1558 Loss: 0.170974001288414
Epoch: 1559 Loss: 0.1707899421453476
Epoch: 1560 Loss: 0.1706976741552353
Epoch: 1561 Loss: 0.17044061422348022
Epoch: 1562 Loss: 0.17020106315612793
Epoch: 1563 Loss: 0.16998393833637238
Epoch: 1564 Loss: 0.17006561160087585
Epoch: 1565 Loss: 0.1696644425392151
Epoch: 1566 Loss: 0.16955628991127014
Epoch: 1567 Loss: 0.1694052815437317
Epoch: 1568 Loss: 0.16905279457569122
Epoch: 1569 Loss: 0.1688433289527893
Epoch: 1570 Loss: 0.16854116320610046
Epoch: 1571 Loss: 0.1683998852968216
Epoch: 1572 Loss: 0.16812336444854736
Epoch: 1573 Loss: 0.1682172417640686
Epoch: 1574 Loss: 0.16785866022109985
Epoch: 1575 Loss: 0.16768699884414673
Epoch: 1576 Loss: 0.16734468936920166
Epoch: 1577 Loss: 0.1673048585653305
Epoch: 1578 Loss: 0.16701260209083557
Epoch: 1579 Loss: 0.1668619066476822
Epoch: 1580 Loss: 0.16660454869270325
Epoch: 1581 Loss: 0.16660389304161072
Epoch: 1582 Loss: 0.16632875800132751
Epoch: 1583 Loss: 0.1662718504667282
Epoch: 1584 Loss: 0.16590376198291779
Epoch: 1585 Loss: 0.16582094132900238
Epoch: 1586 Loss: 0.1654631793498993
Epoch: 1587 Loss: 0.16540034115314484
Epoch: 1588 Loss: 0.16511642932891846
Epoch: 1589 Loss: 0.1648903638124466
Epoch: 1590 Loss: 0.1647024005651474
Epoch: 1591 Loss: 0.1644829362630844
Epoch: 1592 Loss: 0.16426600515842438
Epoch: 1593 Loss: 0.16408771276474
Epoch: 1594 Loss: 0.16395989060401917
Epoch: 1595 Loss: 0.16372054815292358
Epoch: 1596 Loss: 0.1636071354150772
Epoch: 1597 Loss: 0.16348646581172943
Epoch: 1598 Loss: 0.16320885717868805
Epoch: 1599 Loss: 0.16297084093093872
Epoch: 1600 Loss: 0.16292977333068848
Epoch: 1601 Loss: 0.1625513881444931
Epoch: 1602 Loss: 0.16244959831237793
Epoch: 1603 Loss: 0.16226573288440704
Epoch: 1604 Loss: 0.16224981844425201
Epoch: 1605 Loss: 0.1619248241186142
Epoch: 1606 Loss: 0.1617467701435089
Epoch: 1607 Loss: 0.16141246259212494
Epoch: 1608 Loss: 0.1614566147327423
Epoch: 1609 Loss: 0.1611158549785614
Epoch: 1610 Loss: 0.16087649762630463
Epoch: 1611 Loss: 0.16073447465896606
Epoch: 1612 Loss: 0.16068868339061737
Epoch: 1613 Loss: 0.16055753827095032
Epoch: 1614 Loss: 0.16016283631324768
Epoch: 1615 Loss: 0.16017374396324158
Epoch: 1616 Loss: 0.15982438623905182
Epoch: 1617 Loss: 0.15975937247276306
Epoch: 1618 Loss: 0.15962845087051392
Epoch: 1619 Loss: 0.15948082506656647
Epoch: 1620 Loss: 0.15907950699329376
Epoch: 1621 Loss: 0.15894390642642975
Epoch: 1622 Loss: 0.15880370140075684
Epoch: 1623 Loss: 0.15874558687210083
Epoch: 1624 Loss: 0.1583506166934967
Epoch: 1625 Loss: 0.15822292864322662
Epoch: 1626 Loss: 0.15799181163311005
Epoch: 1627 Loss: 0.15779854357242584
Epoch: 1628 Loss: 0.15775395929813385
Epoch: 1629 Loss: 0.15742723643779755
Epoch: 1630 Loss: 0.15737944841384888
Epoch: 1631 Loss: 0.15715056657791138
Epoch: 1632 Loss: 0.15699411928653717
Epoch: 1633 Loss: 0.156966894865036
Epoch: 1634 Loss: 0.15663760900497437
Epoch: 1635 Loss: 0.15638700127601624
Epoch: 1636 Loss: 0.15637631714344025
Epoch: 1637 Loss: 0.15609680116176605
Epoch: 1638 Loss: 0.15599462389945984
Epoch: 1639 Loss: 0.15571069717407227
Epoch: 1640 Loss: 0.1555345356464386
Epoch: 1641 Loss: 0.15555162727832794
Epoch: 1642 Loss: 0.15533767640590668
Epoch: 1643 Loss: 0.15502724051475525
Epoch: 1644 Loss: 0.1549079567193985
Epoch: 1645 Loss: 0.1547137349843979
Epoch: 1646 Loss: 0.1545967012643814
Epoch: 1647 Loss: 0.15447074174880981
Epoch: 1648 Loss: 0.15412630140781403
Epoch: 1649 Loss: 0.15411730110645294
Epoch: 1650 Loss: 0.15383926033973694
Epoch: 1651 Loss: 0.15374678373336792
Epoch: 1652 Loss: 0.15348972380161285
Epoch: 1653 Loss: 0.1533242017030716
Epoch: 1654 Loss: 0.15327708423137665
Epoch: 1655 Loss: 0.1529129445552826
Epoch: 1656 Loss: 0.15277723968029022
Epoch: 1657 Loss: 0.1525561511516571
Epoch: 1658 Loss: 0.15254151821136475
Epoch: 1659 Loss: 0.15231241285800934
Epoch: 1660 Loss: 0.15221454203128815
Epoch: 1661 Loss: 0.15204878151416779
Epoch: 1662 Loss: 0.15179096162319183
Epoch: 1663 Loss: 0.1515752077102661
Epoch: 1664 Loss: 0.15156304836273193
Epoch: 1665 Loss: 0.1512955278158188
Epoch: 1666 Loss: 0.1511922925710678
Epoch: 1667 Loss: 0.1509328931570053
Epoch: 1668 Loss: 0.15081843733787537
Epoch: 1669 Loss: 0.1506805568933487
Epoch: 1670 Loss: 0.15049801766872406
Epoch: 1671 Loss: 0.1503615826368332
Epoch: 1672 Loss: 0.1500004082918167
Epoch: 1673 Loss: 0.1499672532081604
Epoch: 1674 Loss: 0.14967848360538483
Epoch: 1675 Loss: 0.14951473474502563
Epoch: 1676 Loss: 0.14946284890174866
Epoch: 1677 Loss: 0.1491980254650116
Epoch: 1678 Loss: 0.14904724061489105
Epoch: 1679 Loss: 0.1489732265472412
Epoch: 1680 Loss: 0.14876021444797516
Epoch: 1681 Loss: 0.14859099686145782
Epoch: 1682 Loss: 0.14857587218284607
Epoch: 1683 Loss: 0.14824624359607697
Epoch: 1684 Loss: 0.14800408482551575
Epoch: 1685 Loss: 0.14793595671653748
Epoch: 1686 Loss: 0.14787711203098297
Epoch: 1687 Loss: 0.14761605858802795
Epoch: 1688 Loss: 0.14751935005187988
Epoch: 1689 Loss: 0.14731739461421967
Epoch: 1690 Loss: 0.14709246158599854
Epoch: 1691 Loss: 0.14696350693702698
Epoch: 1692 Loss: 0.14680726826190948
Epoch: 1693 Loss: 0.14666160941123962
Epoch: 1694 Loss: 0.146383598446846
Epoch: 1695 Loss: 0.1462850123643875
Epoch: 1696 Loss: 0.14612597227096558
Epoch: 1697 Loss: 0.1461094468832016
Epoch: 1698 Loss: 0.1457560807466507
Epoch: 1699 Loss: 0.14561401307582855
Epoch: 1700 Loss: 0.14546382427215576
Epoch: 1701 Loss: 0.14532320201396942
Epoch: 1702 Loss: 0.14513294398784637
Epoch: 1703 Loss: 0.14500083029270172
Epoch: 1704 Loss: 0.1448647379875183
Epoch: 1705 Loss: 0.14459463953971863
Epoch: 1706 Loss: 0.1444186568260193
Epoch: 1707 Loss: 0.14437545835971832
Epoch: 1708 Loss: 0.14415892958641052
Epoch: 1709 Loss: 0.1440635770559311
Epoch: 1710 Loss: 0.14385609328746796
Epoch: 1711 Loss: 0.14377622306346893
Epoch: 1712 Loss: 0.1434849500656128
Epoch: 1713 Loss: 0.14332394301891327
Epoch: 1714 Loss: 0.1432017982006073
Epoch: 1715 Loss: 0.14327743649482727
Epoch: 1716 Loss: 0.14285604655742645
Epoch: 1717 Loss: 0.1427922397851944
Epoch: 1718 Loss: 0.14263604581356049
Epoch: 1719 Loss: 0.14236317574977875
Epoch: 1720 Loss: 0.14231404662132263
Epoch: 1721 Loss: 0.1421351283788681
Epoch: 1722 Loss: 0.14198896288871765
Epoch: 1723 Loss: 0.14184969663619995
Epoch: 1724 Loss: 0.14182482659816742
Epoch: 1725 Loss: 0.14158165454864502
Epoch: 1726 Loss: 0.1413475126028061
Epoch: 1727 Loss: 0.1412220448255539
Epoch: 1728 Loss: 0.14103372395038605
Epoch: 1729 Loss: 0.14085105061531067
Epoch: 1730 Loss: 0.1407506912946701
Epoch: 1731 Loss: 0.14058052003383636
Epoch: 1732 Loss: 0.14041736721992493
Epoch: 1733 Loss: 0.14024005830287933
Epoch: 1734 Loss: 0.14012575149536133
Epoch: 1735 Loss: 0.14008530974388123
Epoch: 1736 Loss: 0.13979554176330566
Epoch: 1737 Loss: 0.13972514867782593
Epoch: 1738 Loss: 0.13957402110099792
Epoch: 1739 Loss: 0.1394195556640625
Epoch: 1740 Loss: 0.13918958604335785
Epoch: 1741 Loss: 0.13905197381973267
Epoch: 1742 Loss: 0.13901714980602264
Epoch: 1743 Loss: 0.13877072930335999
Epoch: 1744 Loss: 0.13859975337982178
Epoch: 1745 Loss: 0.1385360062122345
Epoch: 1746 Loss: 0.1384090781211853
Epoch: 1747 Loss: 0.1381085366010666
Epoch: 1748 Loss: 0.13799545168876648
Epoch: 1749 Loss: 0.13798391819000244
Epoch: 1750 Loss: 0.13780561089515686
Epoch: 1751 Loss: 0.13771508634090424
Epoch: 1752 Loss: 0.1374371200799942
Epoch: 1753 Loss: 0.13734988868236542
Epoch: 1754 Loss: 0.1371947079896927
Epoch: 1755 Loss: 0.13695554435253143
Epoch: 1756 Loss: 0.136888787150383
Epoch: 1757 Loss: 0.13672366738319397
Epoch: 1758 Loss: 0.13660715520381927
Epoch: 1759 Loss: 0.13650710880756378
Epoch: 1760 Loss: 0.13638915121555328
Epoch: 1761 Loss: 0.13610638678073883
Epoch: 1762 Loss: 0.13601335883140564
Epoch: 1763 Loss: 0.13581141829490662
Epoch: 1764 Loss: 0.13570083677768707
Epoch: 1765 Loss: 0.135588139295578
Epoch: 1766 Loss: 0.13538463413715363
Epoch: 1767 Loss: 0.13538339734077454
Epoch: 1768 Loss: 0.13512703776359558
Epoch: 1769 Loss: 0.13495057821273804
Epoch: 1770 Loss: 0.13480962812900543
Epoch: 1771 Loss: 0.13471978902816772
Epoch: 1772 Loss: 0.13454113900661469
Epoch: 1773 Loss: 0.13439910113811493
Epoch: 1774 Loss: 0.1343850940465927
Epoch: 1775 Loss: 0.13416053354740143
Epoch: 1776 Loss: 0.13413845002651215
Epoch: 1777 Loss: 0.1337868720293045
Epoch: 1778 Loss: 0.1336926370859146
Epoch: 1779 Loss: 0.13368159532546997
Epoch: 1780 Loss: 0.1334051936864853
Epoch: 1781 Loss: 0.13326407968997955
Epoch: 1782 Loss: 0.13321343064308167
Epoch: 1783 Loss: 0.13297536969184875
Epoch: 1784 Loss: 0.13279813528060913
Epoch: 1785 Loss: 0.13274861872196198
Epoch: 1786 Loss: 0.13271820545196533
Epoch: 1787 Loss: 0.1324637532234192
Epoch: 1788 Loss: 0.13249288499355316
Epoch: 1789 Loss: 0.13217793405056
Epoch: 1790 Loss: 0.1320611834526062
Epoch: 1791 Loss: 0.13185478746891022
Epoch: 1792 Loss: 0.13181594014167786
Epoch: 1793 Loss: 0.1317911297082901
Epoch: 1794 Loss: 0.1314810961484909
Epoch: 1795 Loss: 0.13139896094799042
Epoch: 1796 Loss: 0.13136333227157593
Epoch: 1797 Loss: 0.13107822835445404
Epoch: 1798 Loss: 0.13100303709506989
Epoch: 1799 Loss: 0.13079933822155
Epoch: 1800 Loss: 0.13066905736923218
Epoch: 1801 Loss: 0.13065145909786224
Epoch: 1802 Loss: 0.1304551512002945
Epoch: 1803 Loss: 0.13033351302146912
Epoch: 1804 Loss: 0.13010144233703613
Epoch: 1805 Loss: 0.13006627559661865
Epoch: 1806 Loss: 0.12984517216682434
Epoch: 1807 Loss: 0.1296878606081009
Epoch: 1808 Loss: 0.12957069277763367
Epoch: 1809 Loss: 0.12951333820819855
Epoch: 1810 Loss: 0.12926653027534485
Epoch: 1811 Loss: 0.12922757863998413
Epoch: 1812 Loss: 0.12904049456119537
Epoch: 1813 Loss: 0.12891125679016113
Epoch: 1814 Loss: 0.12881293892860413
Epoch: 1815 Loss: 0.1286529004573822
Epoch: 1816 Loss: 0.12855084240436554
Epoch: 1817 Loss: 0.12840166687965393
Epoch: 1818 Loss: 0.12827256321907043
Epoch: 1819 Loss: 0.12808921933174133
Epoch: 1820 Loss: 0.12790833413600922
Epoch: 1821 Loss: 0.12787947058677673
Epoch: 1822 Loss: 0.12772251665592194
Epoch: 1823 Loss: 0.12754851579666138
Epoch: 1824 Loss: 0.12744024395942688
Epoch: 1825 Loss: 0.12727399170398712
Epoch: 1826 Loss: 0.12716205418109894
Epoch: 1827 Loss: 0.12701232731342316
Epoch: 1828 Loss: 0.12688268721103668
Epoch: 1829 Loss: 0.1267121136188507
Epoch: 1830 Loss: 0.1266305297613144
Epoch: 1831 Loss: 0.12653715908527374
Epoch: 1832 Loss: 0.12638400495052338
Epoch: 1833 Loss: 0.1262470930814743
Epoch: 1834 Loss: 0.12608905136585236
Epoch: 1835 Loss: 0.1259119212627411
Epoch: 1836 Loss: 0.1258097141981125
Epoch: 1837 Loss: 0.12566202878952026
Epoch: 1838 Loss: 0.12551365792751312
Epoch: 1839 Loss: 0.12541693449020386
Epoch: 1840 Loss: 0.12524637579917908
Epoch: 1841 Loss: 0.12510092556476593
Epoch: 1842 Loss: 0.12503163516521454
Epoch: 1843 Loss: 0.12492673099040985
Epoch: 1844 Loss: 0.124727264046669
Epoch: 1845 Loss: 0.12478310614824295
Epoch: 1846 Loss: 0.12455198168754578
Epoch: 1847 Loss: 0.12441881746053696
Epoch: 1848 Loss: 0.12428847700357437
Epoch: 1849 Loss: 0.12410835176706314
Epoch: 1850 Loss: 0.1239861398935318
Epoch: 1851 Loss: 0.12385173887014389
Epoch: 1852 Loss: 0.12377145141363144
Epoch: 1853 Loss: 0.1236242949962616
Epoch: 1854 Loss: 0.12354117631912231
Epoch: 1855 Loss: 0.12334856390953064
Epoch: 1856 Loss: 0.1231936663389206
Epoch: 1857 Loss: 0.12308469414710999
Epoch: 1858 Loss: 0.12291089445352554
Epoch: 1859 Loss: 0.12292198091745377
Epoch: 1860 Loss: 0.12280339747667313
Epoch: 1861 Loss: 0.12263984233140945
Epoch: 1862 Loss: 0.1225123405456543
Epoch: 1863 Loss: 0.12234408408403397
Epoch: 1864 Loss: 0.12225178629159927
Epoch: 1865 Loss: 0.12216237187385559
Epoch: 1866 Loss: 0.12197552621364594
Epoch: 1867 Loss: 0.12181558459997177
Epoch: 1868 Loss: 0.1216960996389389
Epoch: 1869 Loss: 0.12167416512966156
Epoch: 1870 Loss: 0.12147335708141327
Epoch: 1871 Loss: 0.12146195769309998
Epoch: 1872 Loss: 0.12126492708921432
Epoch: 1873 Loss: 0.1210993230342865
Epoch: 1874 Loss: 0.12095418572425842
Epoch: 1875 Loss: 0.12084674090147018
Epoch: 1876 Loss: 0.12071983516216278
Epoch: 1877 Loss: 0.1206333339214325
Epoch: 1878 Loss: 0.12047705054283142
Epoch: 1879 Loss: 0.12029361724853516
Epoch: 1880 Loss: 0.120221808552742
Epoch: 1881 Loss: 0.12011323869228363
Epoch: 1882 Loss: 0.11993498355150223
Epoch: 1883 Loss: 0.11985013633966446
Epoch: 1884 Loss: 0.11974561214447021
Epoch: 1885 Loss: 0.11961737275123596
Epoch: 1886 Loss: 0.11953786760568619
Epoch: 1887 Loss: 0.1193542554974556
Epoch: 1888 Loss: 0.11920824646949768
Epoch: 1889 Loss: 0.11911098659038544
Epoch: 1890 Loss: 0.11899115890264511
Epoch: 1891 Loss: 0.11889884620904922
Epoch: 1892 Loss: 0.11879613995552063
Epoch: 1893 Loss: 0.11869402229785919
Epoch: 1894 Loss: 0.11845385283231735
Epoch: 1895 Loss: 0.11837661266326904
Epoch: 1896 Loss: 0.1182464212179184
Epoch: 1897 Loss: 0.11816021054983139
Epoch: 1898 Loss: 0.11801112443208694
Epoch: 1899 Loss: 0.11788434535264969
Epoch: 1900 Loss: 0.11775706708431244
Epoch: 1901 Loss: 0.11762502789497375
Epoch: 1902 Loss: 0.11752978712320328
Epoch: 1903 Loss: 0.11745695024728775
Epoch: 1904 Loss: 0.11728629469871521
Epoch: 1905 Loss: 0.11714060604572296
Epoch: 1906 Loss: 0.11706984043121338
Epoch: 1907 Loss: 0.11695859581232071
Epoch: 1908 Loss: 0.11683957278728485
Epoch: 1909 Loss: 0.11668764799833298
Epoch: 1910 Loss: 0.11655306071043015
Epoch: 1911 Loss: 0.11644040048122406
Epoch: 1912 Loss: 0.11633754521608353
Epoch: 1913 Loss: 0.11620726436376572
Epoch: 1914 Loss: 0.11608900129795074
Epoch: 1915 Loss: 0.11600608378648758
Epoch: 1916 Loss: 0.1159808486700058
Epoch: 1917 Loss: 0.11574781686067581
Epoch: 1918 Loss: 0.11567644774913788
Epoch: 1919 Loss: 0.11547138541936874
Epoch: 1920 Loss: 0.11541102826595306
Epoch: 1921 Loss: 0.11533945798873901
Epoch: 1922 Loss: 0.11517618596553802
Epoch: 1923 Loss: 0.11510720103979111
Epoch: 1924 Loss: 0.11493878811597824
Epoch: 1925 Loss: 0.114816814661026
Epoch: 1926 Loss: 0.11471976339817047
Epoch: 1927 Loss: 0.11461354792118073
Epoch: 1928 Loss: 0.11446669697761536
Epoch: 1929 Loss: 0.1143643856048584
Epoch: 1930 Loss: 0.11431192606687546
Epoch: 1931 Loss: 0.11427301168441772
Epoch: 1932 Loss: 0.11404070258140564
Epoch: 1933 Loss: 0.11393821239471436
Epoch: 1934 Loss: 0.11382759362459183
Epoch: 1935 Loss: 0.11367521435022354
Epoch: 1936 Loss: 0.11367789655923843
Epoch: 1937 Loss: 0.11353670060634613
Epoch: 1938 Loss: 0.11334721744060516
Epoch: 1939 Loss: 0.11334484815597534
Epoch: 1940 Loss: 0.11314699053764343
Epoch: 1941 Loss: 0.11299651116132736
Epoch: 1942 Loss: 0.11297804117202759
Epoch: 1943 Loss: 0.11288535594940186
Epoch: 1944 Loss: 0.11270208656787872
Epoch: 1945 Loss: 0.11264342814683914
Epoch: 1946 Loss: 0.11253318190574646
Epoch: 1947 Loss: 0.11236753314733505
Epoch: 1948 Loss: 0.11228722333908081
Epoch: 1949 Loss: 0.11216897517442703
Epoch: 1950 Loss: 0.11202578246593475
Epoch: 1951 Loss: 0.11197523772716522
Epoch: 1952 Loss: 0.11188965290784836
Epoch: 1953 Loss: 0.11168999969959259
Epoch: 1954 Loss: 0.1115807518362999
Epoch: 1955 Loss: 0.11151129752397537
Epoch: 1956 Loss: 0.11141305416822433
Epoch: 1957 Loss: 0.11132775247097015
Epoch: 1958 Loss: 0.11115804314613342
Epoch: 1959 Loss: 0.11104945093393326
Epoch: 1960 Loss: 0.1109422892332077
Epoch: 1961 Loss: 0.11085281521081924
Epoch: 1962 Loss: 0.11071117967367172
Epoch: 1963 Loss: 0.11061767488718033
Epoch: 1964 Loss: 0.1105707436800003
Epoch: 1965 Loss: 0.11042249202728271
Epoch: 1966 Loss: 0.11027641594409943
Epoch: 1967 Loss: 0.11022358387708664
Epoch: 1968 Loss: 0.11007228493690491
Epoch: 1969 Loss: 0.10994220525026321
Epoch: 1970 Loss: 0.10986453294754028
Epoch: 1971 Loss: 0.10973560065031052
Epoch: 1972 Loss: 0.10959970206022263
Epoch: 1973 Loss: 0.1095910295844078
Epoch: 1974 Loss: 0.10942838340997696
Epoch: 1975 Loss: 0.10936668515205383
Epoch: 1976 Loss: 0.10921779274940491
Epoch: 1977 Loss: 0.10911060124635696
Epoch: 1978 Loss: 0.10898430645465851
Epoch: 1979 Loss: 0.1088537946343422
Epoch: 1980 Loss: 0.1088808923959732
Epoch: 1981 Loss: 0.10868936032056808
Epoch: 1982 Loss: 0.10859863460063934
Epoch: 1983 Loss: 0.10848880559206009
Epoch: 1984 Loss: 0.1084229052066803
Epoch: 1985 Loss: 0.10826904326677322
Epoch: 1986 Loss: 0.10817060619592667
Epoch: 1987 Loss: 0.1080411821603775
Epoch: 1988 Loss: 0.1079404279589653
Epoch: 1989 Loss: 0.10784762352705002
Epoch: 1990 Loss: 0.10777552425861359
Epoch: 1991 Loss: 0.10763970762491226
Epoch: 1992 Loss: 0.10751279443502426
Epoch: 1993 Loss: 0.10741809010505676
Epoch: 1994 Loss: 0.10736515372991562
Epoch: 1995 Loss: 0.1072632446885109
Epoch: 1996 Loss: 0.10717011988162994
Epoch: 1997 Loss: 0.10706663131713867
Epoch: 1998 Loss: 0.10691700130701065
Epoch: 1999 Loss: 0.10683789849281311
Epoch: 2000 Loss: 0.10679470002651215
Epoch: 2001 Loss: 0.10659222304821014
Epoch: 2002 Loss: 0.10652169585227966
Epoch: 2003 Loss: 0.10642934590578079
Epoch: 2004 Loss: 0.10626722127199173
Epoch: 2005 Loss: 0.1061590313911438
Epoch: 2006 Loss: 0.10611246526241302
Epoch: 2007 Loss: 0.10606466978788376
Epoch: 2008 Loss: 0.10591955482959747
Epoch: 2009 Loss: 0.10581514984369278
Epoch: 2010 Loss: 0.1057160422205925
Epoch: 2011 Loss: 0.10559099912643433
Epoch: 2012 Loss: 0.1055009737610817
Epoch: 2013 Loss: 0.10542398691177368
Epoch: 2014 Loss: 0.10530044883489609
Epoch: 2015 Loss: 0.10519511997699738
Epoch: 2016 Loss: 0.10506658256053925
Epoch: 2017 Loss: 0.1049550324678421
Epoch: 2018 Loss: 0.10491912066936493
Epoch: 2019 Loss: 0.1047520637512207
Epoch: 2020 Loss: 0.10470970720052719
Epoch: 2021 Loss: 0.10455769300460815
Epoch: 2022 Loss: 0.1044408529996872
Epoch: 2023 Loss: 0.10435991734266281
Epoch: 2024 Loss: 0.10425516963005066
Epoch: 2025 Loss: 0.104156993329525
Epoch: 2026 Loss: 0.10404936969280243
Epoch: 2027 Loss: 0.10397803038358688
Epoch: 2028 Loss: 0.10385361313819885
Epoch: 2029 Loss: 0.10378029942512512
Epoch: 2030 Loss: 0.10367383062839508
Epoch: 2031 Loss: 0.10359890013933182
Epoch: 2032 Loss: 0.10344932973384857
Epoch: 2033 Loss: 0.10341449081897736
Epoch: 2034 Loss: 0.10330621153116226
Epoch: 2035 Loss: 0.10323584079742432
Epoch: 2036 Loss: 0.10306476801633835
Epoch: 2037 Loss: 0.10296706110239029
Epoch: 2038 Loss: 0.102882981300354
Epoch: 2039 Loss: 0.10280115157365799
Epoch: 2040 Loss: 0.10268592834472656
Epoch: 2041 Loss: 0.102582186460495
Epoch: 2042 Loss: 0.1025247573852539
Epoch: 2043 Loss: 0.10236751288175583
Epoch: 2044 Loss: 0.10231957584619522
Epoch: 2045 Loss: 0.10221529006958008
Epoch: 2046 Loss: 0.1021430566906929
Epoch: 2047 Loss: 0.10199624300003052
Epoch: 2048 Loss: 0.10192997753620148
Epoch: 2049 Loss: 0.10181800276041031
Epoch: 2050 Loss: 0.10171862691640854
Epoch: 2051 Loss: 0.10169068723917007
Epoch: 2052 Loss: 0.10151165723800659
Epoch: 2053 Loss: 0.1014205664396286
Epoch: 2054 Loss: 0.10133278369903564
Epoch: 2055 Loss: 0.10126534104347229
Epoch: 2056 Loss: 0.10114878416061401
Epoch: 2057 Loss: 0.10102066397666931
Epoch: 2058 Loss: 0.10092172771692276
Epoch: 2059 Loss: 0.10086124390363693
Epoch: 2060 Loss: 0.10073469579219818
Epoch: 2061 Loss: 0.10065019875764847
Epoch: 2062 Loss: 0.100581094622612
Epoch: 2063 Loss: 0.10048186033964157
Epoch: 2064 Loss: 0.10040169209241867
Epoch: 2065 Loss: 0.10030969977378845
Epoch: 2066 Loss: 0.10023624449968338
Epoch: 2067 Loss: 0.10007471591234207
Epoch: 2068 Loss: 0.1000233143568039
Epoch: 2069 Loss: 0.09993510693311691
Epoch: 2070 Loss: 0.09984413534402847
Epoch: 2071 Loss: 0.09975647181272507
Epoch: 2072 Loss: 0.09962370246648788
Epoch: 2073 Loss: 0.0995517447590828
Epoch: 2074 Loss: 0.09947104752063751
Epoch: 2075 Loss: 0.09936487674713135
Epoch: 2076 Loss: 0.09926487505435944
Epoch: 2077 Loss: 0.09916277974843979
Epoch: 2078 Loss: 0.09908326715230942
Epoch: 2079 Loss: 0.09901021420955658
Epoch: 2080 Loss: 0.09890663623809814
Epoch: 2081 Loss: 0.09877274185419083
Epoch: 2082 Loss: 0.09872496873140335
Epoch: 2083 Loss: 0.09859809279441833
Epoch: 2084 Loss: 0.09850993007421494
Epoch: 2085 Loss: 0.09841782599687576
Epoch: 2086 Loss: 0.09841202944517136
Epoch: 2087 Loss: 0.09821964055299759
Epoch: 2088 Loss: 0.0981239378452301
Epoch: 2089 Loss: 0.0980166345834732
Epoch: 2090 Loss: 0.0979803204536438
Epoch: 2091 Loss: 0.09784131497144699
Epoch: 2092 Loss: 0.09776399284601212
Epoch: 2093 Loss: 0.09767865389585495
Epoch: 2094 Loss: 0.09762763977050781
Epoch: 2095 Loss: 0.09749798476696014
Epoch: 2096 Loss: 0.09739012271165848
Epoch: 2097 Loss: 0.09729084372520447
Epoch: 2098 Loss: 0.09725768864154816
Epoch: 2099 Loss: 0.09714283049106598
Epoch: 2100 Loss: 0.0970875471830368
Epoch: 2101 Loss: 0.09696796536445618
Epoch: 2102 Loss: 0.09690707921981812
Epoch: 2103 Loss: 0.09680388122797012
Epoch: 2104 Loss: 0.09665168076753616
Epoch: 2105 Loss: 0.09658940136432648
Epoch: 2106 Loss: 0.09648172557353973
Epoch: 2107 Loss: 0.09644608944654465
Epoch: 2108 Loss: 0.09633959829807281
Epoch: 2109 Loss: 0.09626360982656479
Epoch: 2110 Loss: 0.09616579860448837
Epoch: 2111 Loss: 0.09604636579751968
Epoch: 2112 Loss: 0.0959528386592865
Epoch: 2113 Loss: 0.09591646492481232
Epoch: 2114 Loss: 0.09582497179508209
Epoch: 2115 Loss: 0.09572059661149979
Epoch: 2116 Loss: 0.09564664214849472
Epoch: 2117 Loss: 0.0955316349864006
Epoch: 2118 Loss: 0.09544237703084946
Epoch: 2119 Loss: 0.0953526571393013
Epoch: 2120 Loss: 0.09523637592792511
Epoch: 2121 Loss: 0.09517533332109451
Epoch: 2122 Loss: 0.0950675755739212
Epoch: 2123 Loss: 0.09500633925199509
Epoch: 2124 Loss: 0.09491930902004242
Epoch: 2125 Loss: 0.09483060240745544
Epoch: 2126 Loss: 0.09469522535800934
Epoch: 2127 Loss: 0.09463244676589966
Epoch: 2128 Loss: 0.09458710253238678
Epoch: 2129 Loss: 0.09451965242624283
Epoch: 2130 Loss: 0.09438461810350418
Epoch: 2131 Loss: 0.09428728371858597
Epoch: 2132 Loss: 0.09420580416917801
Epoch: 2133 Loss: 0.09415172040462494
Epoch: 2134 Loss: 0.09402254223823547
Epoch: 2135 Loss: 0.09394346922636032
Epoch: 2136 Loss: 0.09388212859630585
Epoch: 2137 Loss: 0.09381383657455444
Epoch: 2138 Loss: 0.09370091557502747
Epoch: 2139 Loss: 0.09365620464086533
Epoch: 2140 Loss: 0.09356169402599335
Epoch: 2141 Loss: 0.09346377104520798
Epoch: 2142 Loss: 0.09336166083812714
Epoch: 2143 Loss: 0.0932881087064743
Epoch: 2144 Loss: 0.0931972786784172
Epoch: 2145 Loss: 0.09310869872570038
Epoch: 2146 Loss: 0.0930827185511589
Epoch: 2147 Loss: 0.09291618317365646
Epoch: 2148 Loss: 0.09283851832151413
Epoch: 2149 Loss: 0.09277468919754028
Epoch: 2150 Loss: 0.09266781806945801
Epoch: 2151 Loss: 0.09260745346546173
Epoch: 2152 Loss: 0.09253526479005814
Epoch: 2153 Loss: 0.09242510795593262
Epoch: 2154 Loss: 0.09235614538192749
Epoch: 2155 Loss: 0.09228340536355972
Epoch: 2156 Loss: 0.09221258759498596
Epoch: 2157 Loss: 0.09213757514953613
Epoch: 2158 Loss: 0.09200628846883774
Epoch: 2159 Loss: 0.09191526472568512
Epoch: 2160 Loss: 0.09184779971837997
Epoch: 2161 Loss: 0.09180781245231628
Epoch: 2162 Loss: 0.09172119945287704
Epoch: 2163 Loss: 0.09163183718919754
Epoch: 2164 Loss: 0.09150982648134232
Epoch: 2165 Loss: 0.09143935889005661
Epoch: 2166 Loss: 0.09140034765005112
Epoch: 2167 Loss: 0.09131721407175064
Epoch: 2168 Loss: 0.09119991213083267
Epoch: 2169 Loss: 0.09110066294670105
Epoch: 2170 Loss: 0.09102025628089905
Epoch: 2171 Loss: 0.09093812853097916
Epoch: 2172 Loss: 0.09084200114011765
Epoch: 2173 Loss: 0.09078548103570938
Epoch: 2174 Loss: 0.09068603068590164
Epoch: 2175 Loss: 0.09064560383558273
Epoch: 2176 Loss: 0.09055223315954208
Epoch: 2177 Loss: 0.0904460996389389
Epoch: 2178 Loss: 0.09036149084568024
Epoch: 2179 Loss: 0.09028517454862595
Epoch: 2180 Loss: 0.09018569439649582
Epoch: 2181 Loss: 0.09013620764017105
Epoch: 2182 Loss: 0.09008345007896423
Epoch: 2183 Loss: 0.08998937159776688
Epoch: 2184 Loss: 0.08990585803985596
Epoch: 2185 Loss: 0.08982106298208237
Epoch: 2186 Loss: 0.08973876386880875
Epoch: 2187 Loss: 0.08962363749742508
Epoch: 2188 Loss: 0.08956072479486465
Epoch: 2189 Loss: 0.08951772004365921
Epoch: 2190 Loss: 0.08942338079214096
Epoch: 2191 Loss: 0.08931812644004822
Epoch: 2192 Loss: 0.08927950263023376
Epoch: 2193 Loss: 0.08913510292768478
Epoch: 2194 Loss: 0.08906716108322144
Epoch: 2195 Loss: 0.08895441144704819
Epoch: 2196 Loss: 0.08889088034629822
Epoch: 2197 Loss: 0.08885308355093002
Epoch: 2198 Loss: 0.08873679488897324
Epoch: 2199 Loss: 0.08863326907157898
Epoch: 2200 Loss: 0.08859852701425552
Epoch: 2201 Loss: 0.08850644528865814
Epoch: 2202 Loss: 0.08842436224222183
Epoch: 2203 Loss: 0.0883386880159378
Epoch: 2204 Loss: 0.08826980739831924
Epoch: 2205 Loss: 0.08820434659719467
Epoch: 2206 Loss: 0.08816275745630264
Epoch: 2207 Loss: 0.08807873725891113
Epoch: 2208 Loss: 0.08795307576656342
Epoch: 2209 Loss: 0.08788109570741653
Epoch: 2210 Loss: 0.08779435604810715
Epoch: 2211 Loss: 0.08775761723518372
Epoch: 2212 Loss: 0.08764234185218811
Epoch: 2213 Loss: 0.08757742494344711
Epoch: 2214 Loss: 0.08752166479825974
Epoch: 2215 Loss: 0.08739137649536133
Epoch: 2216 Loss: 0.08733712136745453
Epoch: 2217 Loss: 0.08724997192621231
Epoch: 2218 Loss: 0.08717844635248184
Epoch: 2219 Loss: 0.0871160551905632
Epoch: 2220 Loss: 0.08701959252357483
Epoch: 2221 Loss: 0.08696096390485764
Epoch: 2222 Loss: 0.08686553686857224
Epoch: 2223 Loss: 0.08674152195453644
Epoch: 2224 Loss: 0.08671766519546509
Epoch: 2225 Loss: 0.08664220571517944
Epoch: 2226 Loss: 0.08658139407634735
Epoch: 2227 Loss: 0.08647236973047256
Epoch: 2228 Loss: 0.08637700974941254
Epoch: 2229 Loss: 0.0863063856959343
Epoch: 2230 Loss: 0.08622874319553375
Epoch: 2231 Loss: 0.08616692572832108
Epoch: 2232 Loss: 0.08607974648475647
Epoch: 2233 Loss: 0.08606453239917755
Epoch: 2234 Loss: 0.08593907207250595
Epoch: 2235 Loss: 0.085845448076725
Epoch: 2236 Loss: 0.08577306568622589
Epoch: 2237 Loss: 0.08576515316963196
Epoch: 2238 Loss: 0.08564328402280807
Epoch: 2239 Loss: 0.08555179834365845
Epoch: 2240 Loss: 0.08551768958568573
Epoch: 2241 Loss: 0.08541731536388397
Epoch: 2242 Loss: 0.08532224595546722
Epoch: 2243 Loss: 0.08527512848377228
Epoch: 2244 Loss: 0.08521264046430588
Epoch: 2245 Loss: 0.08512956649065018
Epoch: 2246 Loss: 0.08503040671348572
Epoch: 2247 Loss: 0.0850280225276947
Epoch: 2248 Loss: 0.08489899337291718
Epoch: 2249 Loss: 0.08482563495635986
Epoch: 2250 Loss: 0.0847184807062149
Epoch: 2251 Loss: 0.0846489742398262
Epoch: 2252 Loss: 0.08454969525337219
Epoch: 2253 Loss: 0.08452846854925156
Epoch: 2254 Loss: 0.08444422483444214
Epoch: 2255 Loss: 0.08434681594371796
Epoch: 2256 Loss: 0.08427342772483826
Epoch: 2257 Loss: 0.08424940705299377
Epoch: 2258 Loss: 0.08415365219116211
Epoch: 2259 Loss: 0.08408110588788986
Epoch: 2260 Loss: 0.08396535366773605
Epoch: 2261 Loss: 0.08389606326818466
Epoch: 2262 Loss: 0.0838514193892479
Epoch: 2263 Loss: 0.08379063755273819
Epoch: 2264 Loss: 0.0836702361702919
Epoch: 2265 Loss: 0.08361810445785522
Epoch: 2266 Loss: 0.08357686549425125
Epoch: 2267 Loss: 0.0834721028804779
Epoch: 2268 Loss: 0.08339298516511917
Epoch: 2269 Loss: 0.08338601142168045
Epoch: 2270 Loss: 0.08325421065092087
Epoch: 2271 Loss: 0.08322778344154358
Epoch: 2272 Loss: 0.08310550451278687
Epoch: 2273 Loss: 0.08304589986801147
Epoch: 2274 Loss: 0.08296995609998703
Epoch: 2275 Loss: 0.08287893980741501
Epoch: 2276 Loss: 0.08286213129758835
Epoch: 2277 Loss: 0.08276478946208954
Epoch: 2278 Loss: 0.08266671001911163
Epoch: 2279 Loss: 0.08260155469179153
Epoch: 2280 Loss: 0.08253110200166702
Epoch: 2281 Loss: 0.08247978240251541
Epoch: 2282 Loss: 0.08238370716571808
Epoch: 2283 Loss: 0.08235248923301697
Epoch: 2284 Loss: 0.08226745575666428
Epoch: 2285 Loss: 0.08215635269880295
Epoch: 2286 Loss: 0.08210819214582443
Epoch: 2287 Loss: 0.08209440112113953
Epoch: 2288 Loss: 0.08198744803667068
Epoch: 2289 Loss: 0.08192265778779984
Epoch: 2290 Loss: 0.08185756206512451
Epoch: 2291 Loss: 0.08178235590457916
Epoch: 2292 Loss: 0.08168216794729233
Epoch: 2293 Loss: 0.08164133131504059
Epoch: 2294 Loss: 0.08157750964164734
Epoch: 2295 Loss: 0.0815228745341301
Epoch: 2296 Loss: 0.0814356729388237
Epoch: 2297 Loss: 0.08138297498226166
Epoch: 2298 Loss: 0.0812825933098793
Epoch: 2299 Loss: 0.08120613545179367
Epoch: 2300 Loss: 0.08112846314907074
Epoch: 2301 Loss: 0.0810760036110878
Epoch: 2302 Loss: 0.08100131154060364
Epoch: 2303 Loss: 0.08093857765197754
Epoch: 2304 Loss: 0.08086719363927841
Epoch: 2305 Loss: 0.08082593232393265
Epoch: 2306 Loss: 0.08073696494102478
Epoch: 2307 Loss: 0.08064678311347961
Epoch: 2308 Loss: 0.0805976465344429
Epoch: 2309 Loss: 0.08049928396940231
Epoch: 2310 Loss: 0.08045752346515656
Epoch: 2311 Loss: 0.08038613945245743
Epoch: 2312 Loss: 0.08030335605144501
Epoch: 2313 Loss: 0.08021630346775055
Epoch: 2314 Loss: 0.08020414412021637
Epoch: 2315 Loss: 0.08013453334569931
Epoch: 2316 Loss: 0.08007435500621796
Epoch: 2317 Loss: 0.07999584078788757
Epoch: 2318 Loss: 0.07992268353700638
Epoch: 2319 Loss: 0.07985880970954895
Epoch: 2320 Loss: 0.07980941981077194
Epoch: 2321 Loss: 0.07970906049013138
Epoch: 2322 Loss: 0.07964103668928146
Epoch: 2323 Loss: 0.07957153767347336
Epoch: 2324 Loss: 0.07949313521385193
Epoch: 2325 Loss: 0.07946658879518509
Epoch: 2326 Loss: 0.07936737686395645
Epoch: 2327 Loss: 0.07929673045873642
Epoch: 2328 Loss: 0.07921301573514938
Epoch: 2329 Loss: 0.07918848842382431
Epoch: 2330 Loss: 0.07912496477365494
Epoch: 2331 Loss: 0.07903183251619339
Epoch: 2332 Loss: 0.07893920689821243
Epoch: 2333 Loss: 0.07887893170118332
Epoch: 2334 Loss: 0.0788324698805809
Epoch: 2335 Loss: 0.07877783477306366
Epoch: 2336 Loss: 0.0787152498960495
Epoch: 2337 Loss: 0.0786612257361412
Epoch: 2338 Loss: 0.07858183979988098
Epoch: 2339 Loss: 0.07853545248508453
Epoch: 2340 Loss: 0.0784636065363884
Epoch: 2341 Loss: 0.07839196920394897
Epoch: 2342 Loss: 0.078341543674469
Epoch: 2343 Loss: 0.07825105637311935
Epoch: 2344 Loss: 0.07818643003702164
Epoch: 2345 Loss: 0.07809766381978989
Epoch: 2346 Loss: 0.07805539667606354
Epoch: 2347 Loss: 0.07798297703266144
Epoch: 2348 Loss: 0.0779014378786087
Epoch: 2349 Loss: 0.07786702364683151
Epoch: 2350 Loss: 0.07779930531978607
Epoch: 2351 Loss: 0.0777028352022171
Epoch: 2352 Loss: 0.07764233648777008
Epoch: 2353 Loss: 0.07758776843547821
Epoch: 2354 Loss: 0.0775226280093193
Epoch: 2355 Loss: 0.07746265828609467
Epoch: 2356 Loss: 0.07745172083377838
Epoch: 2357 Loss: 0.07735026627779007
Epoch: 2358 Loss: 0.07728825509548187
Epoch: 2359 Loss: 0.07718988507986069
Epoch: 2360 Loss: 0.07714246213436127
Epoch: 2361 Loss: 0.07705962657928467
Epoch: 2362 Loss: 0.07701323926448822
Epoch: 2363 Loss: 0.07693060487508774
Epoch: 2364 Loss: 0.07691095024347305
Epoch: 2365 Loss: 0.07682934403419495
Epoch: 2366 Loss: 0.07677974551916122
Epoch: 2367 Loss: 0.07672087103128433
Epoch: 2368 Loss: 0.07663559913635254
Epoch: 2369 Loss: 0.07654618471860886
Epoch: 2370 Loss: 0.07652487605810165
Epoch: 2371 Loss: 0.07645121961832047
Epoch: 2372 Loss: 0.07640010863542557
Epoch: 2373 Loss: 0.07630805671215057
Epoch: 2374 Loss: 0.07625026255846024
Epoch: 2375 Loss: 0.0761948972940445
Epoch: 2376 Loss: 0.07610973715782166
Epoch: 2377 Loss: 0.07605762779712677
Epoch: 2378 Loss: 0.07600918412208557
Epoch: 2379 Loss: 0.07592495530843735
Epoch: 2380 Loss: 0.07587933540344238
Epoch: 2381 Loss: 0.07579156756401062
Epoch: 2382 Loss: 0.07574554532766342
Epoch: 2383 Loss: 0.07566415518522263
Epoch: 2384 Loss: 0.07563424855470657
Epoch: 2385 Loss: 0.07556658238172531
Epoch: 2386 Loss: 0.07550539076328278
Epoch: 2387 Loss: 0.07543444633483887
Epoch: 2388 Loss: 0.07535309344530106
Epoch: 2389 Loss: 0.07531074434518814
Epoch: 2390 Loss: 0.07525628805160522
Epoch: 2391 Loss: 0.07517553120851517
Epoch: 2392 Loss: 0.07511766999959946
Epoch: 2393 Loss: 0.07509325444698334
Epoch: 2394 Loss: 0.0750112384557724
Epoch: 2395 Loss: 0.07494669407606125
Epoch: 2396 Loss: 0.07489731162786484
Epoch: 2397 Loss: 0.07486654072999954
Epoch: 2398 Loss: 0.07476581633090973
Epoch: 2399 Loss: 0.0746764987707138
Epoch: 2400 Loss: 0.07466019690036774
Epoch: 2401 Loss: 0.07459446042776108
Epoch: 2402 Loss: 0.074539415538311
Epoch: 2403 Loss: 0.074456125497818
Epoch: 2404 Loss: 0.07440043985843658
Epoch: 2405 Loss: 0.07433312386274338
Epoch: 2406 Loss: 0.07430850714445114
Epoch: 2407 Loss: 0.07423742115497589
Epoch: 2408 Loss: 0.07417836785316467
Epoch: 2409 Loss: 0.07409977167844772
Epoch: 2410 Loss: 0.07402809709310532
Epoch: 2411 Loss: 0.07397613674402237
Epoch: 2412 Loss: 0.07392329722642899
Epoch: 2413 Loss: 0.07385672628879547
Epoch: 2414 Loss: 0.07380837947130203
Epoch: 2415 Loss: 0.07373137772083282
Epoch: 2416 Loss: 0.07367714494466782
Epoch: 2417 Loss: 0.07361303269863129
Epoch: 2418 Loss: 0.07356761395931244
Epoch: 2419 Loss: 0.07348552346229553
Epoch: 2420 Loss: 0.07343237847089767
Epoch: 2421 Loss: 0.0733712762594223
Epoch: 2422 Loss: 0.07330702990293503
Epoch: 2423 Loss: 0.07324869930744171
Epoch: 2424 Loss: 0.07317870855331421
Epoch: 2425 Loss: 0.073150634765625
Epoch: 2426 Loss: 0.07309627532958984
Epoch: 2427 Loss: 0.07302732765674591
Epoch: 2428 Loss: 0.07297743856906891
Epoch: 2429 Loss: 0.0728965699672699
Epoch: 2430 Loss: 0.07283142954111099
Epoch: 2431 Loss: 0.0728122815489769
Epoch: 2432 Loss: 0.07275380194187164
Epoch: 2433 Loss: 0.07270123809576035
Epoch: 2434 Loss: 0.07260041683912277
Epoch: 2435 Loss: 0.07253582030534744
Epoch: 2436 Loss: 0.07250886410474777
Epoch: 2437 Loss: 0.07244745641946793
Epoch: 2438 Loss: 0.07239752262830734
Epoch: 2439 Loss: 0.0723332092165947
Epoch: 2440 Loss: 0.07227439433336258
Epoch: 2441 Loss: 0.07219666987657547
Epoch: 2442 Loss: 0.07213274389505386
Epoch: 2443 Loss: 0.07208090275526047
Epoch: 2444 Loss: 0.07204288989305496
Epoch: 2445 Loss: 0.07196038216352463
Epoch: 2446 Loss: 0.07195032387971878
Epoch: 2447 Loss: 0.07186751812696457
Epoch: 2448 Loss: 0.07179092615842819
Epoch: 2449 Loss: 0.0717468112707138
Epoch: 2450 Loss: 0.07168734818696976
Epoch: 2451 Loss: 0.07160551100969315
Epoch: 2452 Loss: 0.07158473879098892
Epoch: 2453 Loss: 0.07149647921323776
Epoch: 2454 Loss: 0.07144468277692795
Epoch: 2455 Loss: 0.07140383869409561
Epoch: 2456 Loss: 0.07136744260787964
Epoch: 2457 Loss: 0.07129248231649399
Epoch: 2458 Loss: 0.07126247137784958
Epoch: 2459 Loss: 0.07118475437164307
Epoch: 2460 Loss: 0.0711170881986618
Epoch: 2461 Loss: 0.07105211913585663
Epoch: 2462 Loss: 0.07101619988679886
Epoch: 2463 Loss: 0.07095994055271149
Epoch: 2464 Loss: 0.07092980295419693
Epoch: 2465 Loss: 0.07083051651716232
Epoch: 2466 Loss: 0.07078038156032562
Epoch: 2467 Loss: 0.07070786505937576
Epoch: 2468 Loss: 0.070661760866642
Epoch: 2469 Loss: 0.07061847299337387
Epoch: 2470 Loss: 0.0705382227897644
Epoch: 2471 Loss: 0.07047773152589798
Epoch: 2472 Loss: 0.07045036554336548
Epoch: 2473 Loss: 0.07039777934551239
Epoch: 2474 Loss: 0.07030735909938812
Epoch: 2475 Loss: 0.07028127461671829
Epoch: 2476 Loss: 0.07022996246814728
Epoch: 2477 Loss: 0.07017912715673447
Epoch: 2478 Loss: 0.07008257508277893
Epoch: 2479 Loss: 0.0700400248169899
Epoch: 2480 Loss: 0.06999443471431732
Epoch: 2481 Loss: 0.06995398551225662
Epoch: 2482 Loss: 0.06989523023366928
Epoch: 2483 Loss: 0.06982523202896118
Epoch: 2484 Loss: 0.06975594162940979
Epoch: 2485 Loss: 0.06970612704753876
Epoch: 2486 Loss: 0.0696578323841095
Epoch: 2487 Loss: 0.06963413953781128
Epoch: 2488 Loss: 0.06958947330713272
Epoch: 2489 Loss: 0.06948834657669067
Epoch: 2490 Loss: 0.06941720843315125
Epoch: 2491 Loss: 0.06937417387962341
Epoch: 2492 Loss: 0.06935182958841324
Epoch: 2493 Loss: 0.06931159645318985
Epoch: 2494 Loss: 0.06923182308673859
Epoch: 2495 Loss: 0.06918370723724365
Epoch: 2496 Loss: 0.0691179484128952
Epoch: 2497 Loss: 0.0690484270453453
Epoch: 2498 Loss: 0.06900489330291748
Epoch: 2499 Loss: 0.06896069645881653
Epoch: 2500 Loss: 0.06891816109418869
Epoch: 2501 Loss: 0.06888788938522339
Epoch: 2502 Loss: 0.06882263720035553
Epoch: 2503 Loss: 0.06876064091920853
Epoch: 2504 Loss: 0.06867597252130508
Epoch: 2505 Loss: 0.06863132864236832
Epoch: 2506 Loss: 0.06859921663999557
Epoch: 2507 Loss: 0.06853584200143814
Epoch: 2508 Loss: 0.06845786422491074
Epoch: 2509 Loss: 0.06843075156211853
Epoch: 2510 Loss: 0.06839855760335922
Epoch: 2511 Loss: 0.06833107024431229
Epoch: 2512 Loss: 0.06825204193592072
Epoch: 2513 Loss: 0.06821763515472412
Epoch: 2514 Loss: 0.06814270466566086
Epoch: 2515 Loss: 0.06808286160230637
Epoch: 2516 Loss: 0.06805308163166046
Epoch: 2517 Loss: 0.0679919496178627
Epoch: 2518 Loss: 0.06798584759235382
Epoch: 2519 Loss: 0.06792671978473663
Epoch: 2520 Loss: 0.06783515214920044
Epoch: 2521 Loss: 0.06778591871261597
Epoch: 2522 Loss: 0.0677621066570282
Epoch: 2523 Loss: 0.06770309060811996
Epoch: 2524 Loss: 0.06763717532157898
Epoch: 2525 Loss: 0.0675772950053215
Epoch: 2526 Loss: 0.06752657145261765
Epoch: 2527 Loss: 0.06746610999107361
Epoch: 2528 Loss: 0.06743419915437698
Epoch: 2529 Loss: 0.06735303997993469
Epoch: 2530 Loss: 0.06731480360031128
Epoch: 2531 Loss: 0.06726504862308502
Epoch: 2532 Loss: 0.06722166389226913
Epoch: 2533 Loss: 0.06716956943273544
Epoch: 2534 Loss: 0.0670943632721901
Epoch: 2535 Loss: 0.067076675593853
Epoch: 2536 Loss: 0.06699807941913605
Epoch: 2537 Loss: 0.06698881089687347
Epoch: 2538 Loss: 0.06689808517694473
Epoch: 2539 Loss: 0.0668492391705513
Epoch: 2540 Loss: 0.06682127714157104
Epoch: 2541 Loss: 0.06677623838186264
Epoch: 2542 Loss: 0.06668960303068161
Epoch: 2543 Loss: 0.06664222478866577
Epoch: 2544 Loss: 0.06657886505126953
Epoch: 2545 Loss: 0.06655425578355789
Epoch: 2546 Loss: 0.06649462133646011
Epoch: 2547 Loss: 0.0664580911397934
Epoch: 2548 Loss: 0.06640501320362091
Epoch: 2549 Loss: 0.0663372129201889
Epoch: 2550 Loss: 0.06629693508148193
Epoch: 2551 Loss: 0.06623061001300812
Epoch: 2552 Loss: 0.06620673835277557
Epoch: 2553 Loss: 0.06613391637802124
Epoch: 2554 Loss: 0.06609758734703064
Epoch: 2555 Loss: 0.06604957580566406
Epoch: 2556 Loss: 0.06597438454627991
Epoch: 2557 Loss: 0.0659366175532341
Epoch: 2558 Loss: 0.06591769307851791
Epoch: 2559 Loss: 0.06583324819803238
Epoch: 2560 Loss: 0.06578883528709412
Epoch: 2561 Loss: 0.06574597954750061
Epoch: 2562 Loss: 0.06567329168319702
Epoch: 2563 Loss: 0.06563185900449753
Epoch: 2564 Loss: 0.06558093428611755
Epoch: 2565 Loss: 0.06553348898887634
Epoch: 2566 Loss: 0.0654858648777008
Epoch: 2567 Loss: 0.06544733047485352
Epoch: 2568 Loss: 0.06536108255386353
Epoch: 2569 Loss: 0.06536050885915756
Epoch: 2570 Loss: 0.06530546396970749
Epoch: 2571 Loss: 0.06525067985057831
Epoch: 2572 Loss: 0.0651925578713417
Epoch: 2573 Loss: 0.06516239047050476
Epoch: 2574 Loss: 0.06509016454219818
Epoch: 2575 Loss: 0.06503436714410782
Epoch: 2576 Loss: 0.06498726457357407
Epoch: 2577 Loss: 0.06495805084705353
Epoch: 2578 Loss: 0.06489605456590652
Epoch: 2579 Loss: 0.06485047936439514
Epoch: 2580 Loss: 0.06480264663696289
Epoch: 2581 Loss: 0.06474627554416656
Epoch: 2582 Loss: 0.06468851864337921
Epoch: 2583 Loss: 0.06463690102100372
Epoch: 2584 Loss: 0.06459350138902664
Epoch: 2585 Loss: 0.06457030028104782
Epoch: 2586 Loss: 0.06449958682060242
Epoch: 2587 Loss: 0.06446417421102524
Epoch: 2588 Loss: 0.06438467651605606
Epoch: 2589 Loss: 0.06436409056186676
Epoch: 2590 Loss: 0.0642838254570961
Epoch: 2591 Loss: 0.06422916799783707
Epoch: 2592 Loss: 0.06419423222541809
Epoch: 2593 Loss: 0.0641670897603035
Epoch: 2594 Loss: 0.0640958622097969
Epoch: 2595 Loss: 0.06404901295900345
Epoch: 2596 Loss: 0.06402119994163513
Epoch: 2597 Loss: 0.06396429240703583
Epoch: 2598 Loss: 0.06391546875238419
Epoch: 2599 Loss: 0.0638526976108551
Epoch: 2600 Loss: 0.06381591409444809
Epoch: 2601 Loss: 0.06379085034132004
Epoch: 2602 Loss: 0.06370706856250763
Epoch: 2603 Loss: 0.06366809457540512
Epoch: 2604 Loss: 0.06362450867891312
Epoch: 2605 Loss: 0.06356923282146454
Epoch: 2606 Loss: 0.0635177418589592
Epoch: 2607 Loss: 0.06345070898532867
Epoch: 2608 Loss: 0.06341267377138138
Epoch: 2609 Loss: 0.0633731409907341
Epoch: 2610 Loss: 0.06334030628204346
Epoch: 2611 Loss: 0.06326597929000854
Epoch: 2612 Loss: 0.06327611207962036
Epoch: 2613 Loss: 0.06318957358598709
Epoch: 2614 Loss: 0.06313243508338928
Epoch: 2615 Loss: 0.06308214366436005
Epoch: 2616 Loss: 0.06305338442325592
Epoch: 2617 Loss: 0.06298458576202393
Epoch: 2618 Loss: 0.06292865425348282
Epoch: 2619 Loss: 0.0629005953669548
Epoch: 2620 Loss: 0.06284883618354797
Epoch: 2621 Loss: 0.06283096969127655
Epoch: 2622 Loss: 0.06274883449077606
Epoch: 2623 Loss: 0.06271146237850189
Epoch: 2624 Loss: 0.06266625225543976
Epoch: 2625 Loss: 0.06263934075832367
Epoch: 2626 Loss: 0.0625678077340126
Epoch: 2627 Loss: 0.06251169741153717
Epoch: 2628 Loss: 0.06249598413705826
Epoch: 2629 Loss: 0.06245778873562813
Epoch: 2630 Loss: 0.062409497797489166
Epoch: 2631 Loss: 0.062339723110198975
Epoch: 2632 Loss: 0.0622745156288147
Epoch: 2633 Loss: 0.062245242297649384
Epoch: 2634 Loss: 0.062204401940107346
Epoch: 2635 Loss: 0.062144726514816284
Epoch: 2636 Loss: 0.0621173158288002
Epoch: 2637 Loss: 0.06205807626247406
Epoch: 2638 Loss: 0.06200870871543884
Epoch: 2639 Loss: 0.061966121196746826
Epoch: 2640 Loss: 0.061941105872392654
Epoch: 2641 Loss: 0.06187751516699791
Epoch: 2642 Loss: 0.06182219833135605
Epoch: 2643 Loss: 0.061782173812389374
Epoch: 2644 Loss: 0.06175568699836731
Epoch: 2645 Loss: 0.06168903037905693
Epoch: 2646 Loss: 0.06166060268878937
Epoch: 2647 Loss: 0.06160247325897217
Epoch: 2648 Loss: 0.06156381964683533
Epoch: 2649 Loss: 0.06150216981768608
Epoch: 2650 Loss: 0.061485014855861664
Epoch: 2651 Loss: 0.06141408160328865
Epoch: 2652 Loss: 0.06140431761741638
Epoch: 2653 Loss: 0.06132696196436882
Epoch: 2654 Loss: 0.06130989268422127
Epoch: 2655 Loss: 0.06124952435493469
Epoch: 2656 Loss: 0.061225857585668564
Epoch: 2657 Loss: 0.06114048510789871
Epoch: 2658 Loss: 0.06109336018562317
Epoch: 2659 Loss: 0.06108897179365158
Epoch: 2660 Loss: 0.06103840097784996
Epoch: 2661 Loss: 0.06096860393881798
Epoch: 2662 Loss: 0.06092260405421257
Epoch: 2663 Loss: 0.06087559089064598
Epoch: 2664 Loss: 0.06085134670138359
Epoch: 2665 Loss: 0.06080310791730881
Epoch: 2666 Loss: 0.06074172258377075
Epoch: 2667 Loss: 0.060704320669174194
Epoch: 2668 Loss: 0.06068304553627968
Epoch: 2669 Loss: 0.060623206198215485
Epoch: 2670 Loss: 0.060570064932107925
Epoch: 2671 Loss: 0.060540225356817245
Epoch: 2672 Loss: 0.06047650799155235
Epoch: 2673 Loss: 0.06045063957571983
Epoch: 2674 Loss: 0.06041004881262779
Epoch: 2675 Loss: 0.060343217104673386
Epoch: 2676 Loss: 0.06030864268541336
Epoch: 2677 Loss: 0.06024421751499176
Epoch: 2678 Loss: 0.06021732836961746
Epoch: 2679 Loss: 0.06019248068332672
Epoch: 2680 Loss: 0.060121554881334305
Epoch: 2681 Loss: 0.06011006236076355
Epoch: 2682 Loss: 0.06007082015275955
Epoch: 2683 Loss: 0.06001925468444824
Epoch: 2684 Loss: 0.05997879058122635
Epoch: 2685 Loss: 0.0598970465362072
Epoch: 2686 Loss: 0.05987103283405304
Epoch: 2687 Loss: 0.05986183136701584
Epoch: 2688 Loss: 0.0598067045211792
Epoch: 2689 Loss: 0.05975160747766495
Epoch: 2690 Loss: 0.059698641300201416
Epoch: 2691 Loss: 0.05967537686228752
Epoch: 2692 Loss: 0.05963998660445213
Epoch: 2693 Loss: 0.05956996977329254
Epoch: 2694 Loss: 0.05952898785471916
Epoch: 2695 Loss: 0.05948173254728317
Epoch: 2696 Loss: 0.05944235622882843
Epoch: 2697 Loss: 0.05940937250852585
Epoch: 2698 Loss: 0.059378620237112045
Epoch: 2699 Loss: 0.05931200832128525
Epoch: 2700 Loss: 0.05927768722176552
Epoch: 2701 Loss: 0.05923009291291237
Epoch: 2702 Loss: 0.059211671352386475
Epoch: 2703 Loss: 0.059157468378543854
Epoch: 2704 Loss: 0.059096697717905045
Epoch: 2705 Loss: 0.05907316505908966
Epoch: 2706 Loss: 0.05903499573469162
Epoch: 2707 Loss: 0.05898565798997879
Epoch: 2708 Loss: 0.058938100934028625
Epoch: 2709 Loss: 0.05889474228024483
Epoch: 2710 Loss: 0.05885564163327217
Epoch: 2711 Loss: 0.05883679538965225
Epoch: 2712 Loss: 0.058768097311258316
Epoch: 2713 Loss: 0.05872908979654312
Epoch: 2714 Loss: 0.05870021879673004
Epoch: 2715 Loss: 0.058652713894844055
Epoch: 2716 Loss: 0.05860506743192673
Epoch: 2717 Loss: 0.05857164040207863
Epoch: 2718 Loss: 0.058535266667604446
Epoch: 2719 Loss: 0.05849508196115494
Epoch: 2720 Loss: 0.05843692645430565
Epoch: 2721 Loss: 0.05838842689990997
Epoch: 2722 Loss: 0.05834764987230301
Epoch: 2723 Loss: 0.058316707611083984
Epoch: 2724 Loss: 0.0582730658352375
Epoch: 2725 Loss: 0.058238543570041656
Epoch: 2726 Loss: 0.05818324536085129
Epoch: 2727 Loss: 0.058155979961156845
Epoch: 2728 Loss: 0.058104828000068665
Epoch: 2729 Loss: 0.05805124714970589
Epoch: 2730 Loss: 0.0580420084297657
Epoch: 2731 Loss: 0.057986482977867126
Epoch: 2732 Loss: 0.057952363044023514
Epoch: 2733 Loss: 0.05790286883711815
Epoch: 2734 Loss: 0.057868167757987976
Epoch: 2735 Loss: 0.057843904942274094
Epoch: 2736 Loss: 0.057783134281635284
Epoch: 2737 Loss: 0.05774129554629326
Epoch: 2738 Loss: 0.05770648643374443
Epoch: 2739 Loss: 0.05767818167805672
Epoch: 2740 Loss: 0.05762259289622307
Epoch: 2741 Loss: 0.05761155113577843
Epoch: 2742 Loss: 0.05755509063601494
Epoch: 2743 Loss: 0.05749741569161415
Epoch: 2744 Loss: 0.057476937770843506
Epoch: 2745 Loss: 0.05741410329937935
Epoch: 2746 Loss: 0.05737082287669182
Epoch: 2747 Loss: 0.05735243856906891
Epoch: 2748 Loss: 0.05729744955897331
Epoch: 2749 Loss: 0.0572860985994339
Epoch: 2750 Loss: 0.05723879858851433
Epoch: 2751 Loss: 0.05719394236803055
Epoch: 2752 Loss: 0.057162098586559296
Epoch: 2753 Loss: 0.057108279317617416
Epoch: 2754 Loss: 0.05707378685474396
Epoch: 2755 Loss: 0.05703306198120117
Epoch: 2756 Loss: 0.05698108300566673
Epoch: 2757 Loss: 0.05695032328367233
Epoch: 2758 Loss: 0.056915394961833954
Epoch: 2759 Loss: 0.05687866359949112
Epoch: 2760 Loss: 0.0568205900490284
Epoch: 2761 Loss: 0.05678536742925644
Epoch: 2762 Loss: 0.05675642192363739
Epoch: 2763 Loss: 0.05671272054314613
Epoch: 2764 Loss: 0.05667460337281227
Epoch: 2765 Loss: 0.05664883553981781
Epoch: 2766 Loss: 0.05660640448331833
Epoch: 2767 Loss: 0.0565536692738533
Epoch: 2768 Loss: 0.05652996897697449
Epoch: 2769 Loss: 0.05647912621498108
Epoch: 2770 Loss: 0.05643830448389053
Epoch: 2771 Loss: 0.05640113353729248
Epoch: 2772 Loss: 0.05635659396648407
Epoch: 2773 Loss: 0.05633049085736275
Epoch: 2774 Loss: 0.056273531168699265
Epoch: 2775 Loss: 0.056248363107442856
Epoch: 2776 Loss: 0.05620219558477402
Epoch: 2777 Loss: 0.05615829676389694
Epoch: 2778 Loss: 0.05612485110759735
Epoch: 2779 Loss: 0.05609086528420448
Epoch: 2780 Loss: 0.056046437472105026
Epoch: 2781 Loss: 0.0560067780315876
Epoch: 2782 Loss: 0.05597404018044472
Epoch: 2783 Loss: 0.05593622103333473
Epoch: 2784 Loss: 0.055894408375024796
Epoch: 2785 Loss: 0.05586179718375206
Epoch: 2786 Loss: 0.055834490805864334
Epoch: 2787 Loss: 0.055772315710783005
Epoch: 2788 Loss: 0.055743202567100525
Epoch: 2789 Loss: 0.05570385605096817
Epoch: 2790 Loss: 0.05567058175802231
Epoch: 2791 Loss: 0.05564809590578079
Epoch: 2792 Loss: 0.055622562766075134
Epoch: 2793 Loss: 0.05556652694940567
Epoch: 2794 Loss: 0.05553040653467178
Epoch: 2795 Loss: 0.05548623204231262
Epoch: 2796 Loss: 0.0554375946521759
Epoch: 2797 Loss: 0.055405303835868835
Epoch: 2798 Loss: 0.05538243055343628
Epoch: 2799 Loss: 0.05534893646836281
Epoch: 2800 Loss: 0.05529768392443657
Epoch: 2801 Loss: 0.055260758846998215
Epoch: 2802 Loss: 0.055221401154994965
Epoch: 2803 Loss: 0.055180419236421585
Epoch: 2804 Loss: 0.05515312775969505
Epoch: 2805 Loss: 0.05510864406824112
Epoch: 2806 Loss: 0.05510147288441658
Epoch: 2807 Loss: 0.055049799382686615
Epoch: 2808 Loss: 0.05500393733382225
Epoch: 2809 Loss: 0.054977383464574814
Epoch: 2810 Loss: 0.05491672083735466
Epoch: 2811 Loss: 0.05488744378089905
Epoch: 2812 Loss: 0.05484980717301369
Epoch: 2813 Loss: 0.054813116788864136
Epoch: 2814 Loss: 0.05476623401045799
Epoch: 2815 Loss: 0.05473676696419716
Epoch: 2816 Loss: 0.054696161299943924
Epoch: 2817 Loss: 0.05467034876346588
Epoch: 2818 Loss: 0.05461307242512703
Epoch: 2819 Loss: 0.05460239574313164
Epoch: 2820 Loss: 0.054549187421798706
Epoch: 2821 Loss: 0.054520949721336365
Epoch: 2822 Loss: 0.05449167639017105
Epoch: 2823 Loss: 0.054443977773189545
Epoch: 2824 Loss: 0.05440514162182808
Epoch: 2825 Loss: 0.05439431965351105
Epoch: 2826 Loss: 0.05434781312942505
Epoch: 2827 Loss: 0.05429140478372574
Epoch: 2828 Loss: 0.05424948409199715
Epoch: 2829 Loss: 0.05422559753060341
Epoch: 2830 Loss: 0.05419342592358589
Epoch: 2831 Loss: 0.054170433431863785
Epoch: 2832 Loss: 0.054126638919115067
Epoch: 2833 Loss: 0.0540783628821373
Epoch: 2834 Loss: 0.054057344794273376
Epoch: 2835 Loss: 0.05402343347668648
Epoch: 2836 Loss: 0.05397849529981613
Epoch: 2837 Loss: 0.05395635962486267
Epoch: 2838 Loss: 0.05390017852187157
Epoch: 2839 Loss: 0.0538732074201107
Epoch: 2840 Loss: 0.053821321576833725
Epoch: 2841 Loss: 0.053797535598278046
Epoch: 2842 Loss: 0.05375524237751961
Epoch: 2843 Loss: 0.05373958870768547
Epoch: 2844 Loss: 0.05367468297481537
Epoch: 2845 Loss: 0.0536537803709507
Epoch: 2846 Loss: 0.053614333271980286
Epoch: 2847 Loss: 0.05358261242508888
Epoch: 2848 Loss: 0.05355291813611984
Epoch: 2849 Loss: 0.053506407886743546
Epoch: 2850 Loss: 0.05348169058561325
Epoch: 2851 Loss: 0.053448352962732315
Epoch: 2852 Loss: 0.05340926721692085
Epoch: 2853 Loss: 0.05336001142859459
Epoch: 2854 Loss: 0.0533418133854866
Epoch: 2855 Loss: 0.05328589305281639
Epoch: 2856 Loss: 0.053262438625097275
Epoch: 2857 Loss: 0.0532260462641716
Epoch: 2858 Loss: 0.053176965564489365
Epoch: 2859 Loss: 0.053171247243881226
Epoch: 2860 Loss: 0.05313177406787872
Epoch: 2861 Loss: 0.05309508740901947
Epoch: 2862 Loss: 0.053034164011478424
Epoch: 2863 Loss: 0.0530044287443161
Epoch: 2864 Loss: 0.05297205224633217
Epoch: 2865 Loss: 0.05294720456004143
Epoch: 2866 Loss: 0.052906010299921036
Epoch: 2867 Loss: 0.05287450551986694
Epoch: 2868 Loss: 0.05284874141216278
Epoch: 2869 Loss: 0.05284350365400314
Epoch: 2870 Loss: 0.05281410366296768
Epoch: 2871 Loss: 0.05273978039622307
Epoch: 2872 Loss: 0.05270284414291382
Epoch: 2873 Loss: 0.052669938653707504
Epoch: 2874 Loss: 0.052626874297857285
Epoch: 2875 Loss: 0.05259512737393379
Epoch: 2876 Loss: 0.052565623074769974
Epoch: 2877 Loss: 0.05253357067704201
Epoch: 2878 Loss: 0.05249011144042015
Epoch: 2879 Loss: 0.05246700346469879
Epoch: 2880 Loss: 0.05243048444390297
Epoch: 2881 Loss: 0.05239667370915413
Epoch: 2882 Loss: 0.05236215889453888
Epoch: 2883 Loss: 0.05231984704732895
Epoch: 2884 Loss: 0.052296508103609085
Epoch: 2885 Loss: 0.05228043720126152
Epoch: 2886 Loss: 0.05220639333128929
Epoch: 2887 Loss: 0.05217218026518822
Epoch: 2888 Loss: 0.052142683416604996
Epoch: 2889 Loss: 0.05211348086595535
Epoch: 2890 Loss: 0.052078235894441605
Epoch: 2891 Loss: 0.05207173898816109
Epoch: 2892 Loss: 0.05201557278633118
Epoch: 2893 Loss: 0.05198962986469269
Epoch: 2894 Loss: 0.05197303369641304
Epoch: 2895 Loss: 0.0519399419426918
Epoch: 2896 Loss: 0.05188107490539551
Epoch: 2897 Loss: 0.05185529589653015
Epoch: 2898 Loss: 0.05181379243731499
Epoch: 2899 Loss: 0.051778435707092285
Epoch: 2900 Loss: 0.051742102950811386
Epoch: 2901 Loss: 0.0517115443944931
Epoch: 2902 Loss: 0.05167446285486221
Epoch: 2903 Loss: 0.05167217552661896
Epoch: 2904 Loss: 0.051622942090034485
Epoch: 2905 Loss: 0.05159015581011772
Epoch: 2906 Loss: 0.05155245214700699
Epoch: 2907 Loss: 0.05152035877108574
Epoch: 2908 Loss: 0.051496148109436035
Epoch: 2909 Loss: 0.05144617706537247
Epoch: 2910 Loss: 0.051421284675598145
Epoch: 2911 Loss: 0.05138489231467247
Epoch: 2912 Loss: 0.05134721100330353
Epoch: 2913 Loss: 0.051312755793333054
Epoch: 2914 Loss: 0.05126991868019104
Epoch: 2915 Loss: 0.05124283581972122
Epoch: 2916 Loss: 0.051218606531620026
Epoch: 2917 Loss: 0.05117763578891754
Epoch: 2918 Loss: 0.05115068703889847
Epoch: 2919 Loss: 0.05110682547092438
Epoch: 2920 Loss: 0.051096148788928986
Epoch: 2921 Loss: 0.05105701833963394
Epoch: 2922 Loss: 0.05101208761334419
Epoch: 2923 Loss: 0.051021087914705276
Epoch: 2924 Loss: 0.050953399389982224
Epoch: 2925 Loss: 0.05092230811715126
Epoch: 2926 Loss: 0.05090126395225525
Epoch: 2927 Loss: 0.05086834728717804
Epoch: 2928 Loss: 0.05082331597805023
Epoch: 2929 Loss: 0.050806399434804916
Epoch: 2930 Loss: 0.05075395852327347
Epoch: 2931 Loss: 0.050734758377075195
Epoch: 2932 Loss: 0.05068923532962799
Epoch: 2933 Loss: 0.05066593736410141
Epoch: 2934 Loss: 0.05063559487462044
Epoch: 2935 Loss: 0.050593696534633636
Epoch: 2936 Loss: 0.050575025379657745
Epoch: 2937 Loss: 0.05054575949907303
Epoch: 2938 Loss: 0.050493743270635605
Epoch: 2939 Loss: 0.05046958103775978
Epoch: 2940 Loss: 0.05044661462306976
Epoch: 2941 Loss: 0.050404082983732224
Epoch: 2942 Loss: 0.05038043484091759
Epoch: 2943 Loss: 0.050349246710538864
Epoch: 2944 Loss: 0.05030914396047592
Epoch: 2945 Loss: 0.05027926713228226
Epoch: 2946 Loss: 0.05024553835391998
Epoch: 2947 Loss: 0.05021621286869049
Epoch: 2948 Loss: 0.05018860474228859
Epoch: 2949 Loss: 0.050149813294410706
Epoch: 2950 Loss: 0.0501180961728096
Epoch: 2951 Loss: 0.05008258670568466
Epoch: 2952 Loss: 0.050048839300870895
Epoch: 2953 Loss: 0.05003497749567032
Epoch: 2954 Loss: 0.04999538138508797
Epoch: 2955 Loss: 0.0499623604118824
Epoch: 2956 Loss: 0.04994650557637215
Epoch: 2957 Loss: 0.04988676309585571
Epoch: 2958 Loss: 0.04987155273556709
Epoch: 2959 Loss: 0.049836643040180206
Epoch: 2960 Loss: 0.049808964133262634
Epoch: 2961 Loss: 0.04976543039083481
Epoch: 2962 Loss: 0.049750782549381256
Epoch: 2963 Loss: 0.04972065985202789
Epoch: 2964 Loss: 0.049673616886138916
Epoch: 2965 Loss: 0.049628596752882004
Epoch: 2966 Loss: 0.04963124170899391
Epoch: 2967 Loss: 0.049578674137592316
Epoch: 2968 Loss: 0.04954051226377487
Epoch: 2969 Loss: 0.04952678829431534
Epoch: 2970 Loss: 0.049474988132715225
Epoch: 2971 Loss: 0.049452755600214005
Epoch: 2972 Loss: 0.04944620281457901
Epoch: 2973 Loss: 0.04941100254654884
Epoch: 2974 Loss: 0.0493609756231308
Epoch: 2975 Loss: 0.04933931678533554
Epoch: 2976 Loss: 0.049305956810712814
Epoch: 2977 Loss: 0.049286291003227234
Epoch: 2978 Loss: 0.049246013164520264
Epoch: 2979 Loss: 0.04920892417430878
Epoch: 2980 Loss: 0.04917430132627487
Epoch: 2981 Loss: 0.04916512593626976
Epoch: 2982 Loss: 0.0491081178188324
Epoch: 2983 Loss: 0.049094848334789276
Epoch: 2984 Loss: 0.04905920475721359
Epoch: 2985 Loss: 0.04903990775346756
Epoch: 2986 Loss: 0.04899798706173897
Epoch: 2987 Loss: 0.04895997792482376
Epoch: 2988 Loss: 0.04893988370895386
Epoch: 2989 Loss: 0.04890142008662224
Epoch: 2990 Loss: 0.04888521134853363
Epoch: 2991 Loss: 0.04885001853108406
Epoch: 2992 Loss: 0.04881563410162926
Epoch: 2993 Loss: 0.04878050833940506
Epoch: 2994 Loss: 0.04877137020230293
Epoch: 2995 Loss: 0.04871508851647377
Epoch: 2996 Loss: 0.04867924749851227
Epoch: 2997 Loss: 0.048656485974788666
Epoch: 2998 Loss: 0.04862058535218239
Epoch: 2999 Loss: 0.048598360270261765
Epoch: 3000 Loss: 0.04858449101448059
Epoch: 3001 Loss: 0.04854336008429527
Epoch: 3002 Loss: 0.04850645735859871
Epoch: 3003 Loss: 0.04849047586321831
Epoch: 3004 Loss: 0.0484456829726696
Epoch: 3005 Loss: 0.04841911792755127
Epoch: 3006 Loss: 0.048393070697784424
Epoch: 3007 Loss: 0.04835755378007889
Epoch: 3008 Loss: 0.04832141473889351
Epoch: 3009 Loss: 0.04829562455415726
Epoch: 3010 Loss: 0.04825826361775398
Epoch: 3011 Loss: 0.04823523759841919
Epoch: 3012 Loss: 0.04819357395172119
Epoch: 3013 Loss: 0.0481729581952095
Epoch: 3014 Loss: 0.048156268894672394
Epoch: 3015 Loss: 0.048119060695171356
Epoch: 3016 Loss: 0.048075299710035324
Epoch: 3017 Loss: 0.04805845394730568
Epoch: 3018 Loss: 0.04802084341645241
Epoch: 3019 Loss: 0.0480053573846817
Epoch: 3020 Loss: 0.0479721762239933
Epoch: 3021 Loss: 0.047927141189575195
Epoch: 3022 Loss: 0.04791003838181496
Epoch: 3023 Loss: 0.04790109023451805
Epoch: 3024 Loss: 0.047845132648944855
Epoch: 3025 Loss: 0.0478416308760643
Epoch: 3026 Loss: 0.047803960740566254
Epoch: 3027 Loss: 0.04776262491941452
Epoch: 3028 Loss: 0.04774509370326996
Epoch: 3029 Loss: 0.047717440873384476
Epoch: 3030 Loss: 0.04767245054244995
Epoch: 3031 Loss: 0.0476512610912323
Epoch: 3032 Loss: 0.0476202629506588
Epoch: 3033 Loss: 0.047594502568244934
Epoch: 3034 Loss: 0.047561682760715485
Epoch: 3035 Loss: 0.0475277341902256
Epoch: 3036 Loss: 0.04750990495085716
Epoch: 3037 Loss: 0.047473713755607605
Epoch: 3038 Loss: 0.04744071885943413
Epoch: 3039 Loss: 0.04742381349205971
Epoch: 3040 Loss: 0.04739436134696007
Epoch: 3041 Loss: 0.047360409051179886
Epoch: 3042 Loss: 0.04732690379023552
Epoch: 3043 Loss: 0.0472901277244091
Epoch: 3044 Loss: 0.04728233814239502
Epoch: 3045 Loss: 0.04723787307739258
Epoch: 3046 Loss: 0.04719598591327667
Epoch: 3047 Loss: 0.047192882746458054
Epoch: 3048 Loss: 0.04715772718191147
Epoch: 3049 Loss: 0.04712109640240669
Epoch: 3050 Loss: 0.0470881313085556
Epoch: 3051 Loss: 0.04707195609807968
Epoch: 3052 Loss: 0.04704047739505768
Epoch: 3053 Loss: 0.04701228067278862
Epoch: 3054 Loss: 0.04699838533997536
Epoch: 3055 Loss: 0.046949855983257294
Epoch: 3056 Loss: 0.046938907355070114
Epoch: 3057 Loss: 0.04689579829573631
Epoch: 3058 Loss: 0.046875327825546265
Epoch: 3059 Loss: 0.046841058880090714
Epoch: 3060 Loss: 0.04681095480918884
Epoch: 3061 Loss: 0.046767283231019974
Epoch: 3062 Loss: 0.04675447568297386
Epoch: 3063 Loss: 0.04675602912902832
Epoch: 3064 Loss: 0.04668885096907616
Epoch: 3065 Loss: 0.04666023328900337
Epoch: 3066 Loss: 0.04665452614426613
Epoch: 3067 Loss: 0.04661410301923752
Epoch: 3068 Loss: 0.04657704755663872
Epoch: 3069 Loss: 0.04655999690294266
Epoch: 3070 Loss: 0.04654182493686676
Epoch: 3071 Loss: 0.046518199145793915
Epoch: 3072 Loss: 0.04646534472703934
Epoch: 3073 Loss: 0.046446722000837326
Epoch: 3074 Loss: 0.04641694575548172
Epoch: 3075 Loss: 0.04639381542801857
Epoch: 3076 Loss: 0.04635332524776459
Epoch: 3077 Loss: 0.046327948570251465
Epoch: 3078 Loss: 0.04629765450954437
Epoch: 3079 Loss: 0.046275801956653595
Epoch: 3080 Loss: 0.046239905059337616
Epoch: 3081 Loss: 0.046206749975681305
Epoch: 3082 Loss: 0.04618203639984131
Epoch: 3083 Loss: 0.04619294032454491
Epoch: 3084 Loss: 0.046149127185344696
Epoch: 3085 Loss: 0.04611183702945709
Epoch: 3086 Loss: 0.04608318954706192
Epoch: 3087 Loss: 0.04605424776673317
Epoch: 3088 Loss: 0.0460326187312603
Epoch: 3089 Loss: 0.046020884066820145
Epoch: 3090 Loss: 0.04598530754446983
Epoch: 3091 Loss: 0.04595138132572174
Epoch: 3092 Loss: 0.04592176899313927
Epoch: 3093 Loss: 0.045892439782619476
Epoch: 3094 Loss: 0.04586995020508766
Epoch: 3095 Loss: 0.04583136737346649
Epoch: 3096 Loss: 0.045808181166648865
Epoch: 3097 Loss: 0.045777641236782074
Epoch: 3098 Loss: 0.04575676470994949
Epoch: 3099 Loss: 0.04572982341051102
Epoch: 3100 Loss: 0.04570241644978523
Epoch: 3101 Loss: 0.04566894471645355
Epoch: 3102 Loss: 0.04564497247338295
Epoch: 3103 Loss: 0.0456206277012825
Epoch: 3104 Loss: 0.045593634247779846
Epoch: 3105 Loss: 0.0455656461417675
Epoch: 3106 Loss: 0.04556083306670189
Epoch: 3107 Loss: 0.045506589114665985
Epoch: 3108 Loss: 0.04549550637602806
Epoch: 3109 Loss: 0.04545895755290985
Epoch: 3110 Loss: 0.045428596436977386
Epoch: 3111 Loss: 0.045408643782138824
Epoch: 3112 Loss: 0.0453694723546505
Epoch: 3113 Loss: 0.045356690883636475
Epoch: 3114 Loss: 0.045326054096221924
Epoch: 3115 Loss: 0.04530875012278557
Epoch: 3116 Loss: 0.04526376351714134
Epoch: 3117 Loss: 0.045244526118040085
Epoch: 3118 Loss: 0.04520959407091141
Epoch: 3119 Loss: 0.04520242288708687
Epoch: 3120 Loss: 0.04519931972026825
Epoch: 3121 Loss: 0.045147329568862915
Epoch: 3122 Loss: 0.04512706398963928
Epoch: 3123 Loss: 0.04508139565587044
Epoch: 3124 Loss: 0.04505326226353645
Epoch: 3125 Loss: 0.04504227638244629
Epoch: 3126 Loss: 0.045023489743471146
Epoch: 3127 Loss: 0.0449855662882328
Epoch: 3128 Loss: 0.04496899992227554
Epoch: 3129 Loss: 0.04492145776748657
Epoch: 3130 Loss: 0.04489689692854881
Epoch: 3131 Loss: 0.044882260262966156
Epoch: 3132 Loss: 0.04484784975647926
Epoch: 3133 Loss: 0.04481694474816322
Epoch: 3134 Loss: 0.04479263350367546
Epoch: 3135 Loss: 0.04478239640593529
Epoch: 3136 Loss: 0.044738758355379105
Epoch: 3137 Loss: 0.044714704155921936
Epoch: 3138 Loss: 0.04469815641641617
Epoch: 3139 Loss: 0.04466923326253891
Epoch: 3140 Loss: 0.04463541880249977
Epoch: 3141 Loss: 0.04460858181118965
Epoch: 3142 Loss: 0.0446004644036293
Epoch: 3143 Loss: 0.044568274170160294
Epoch: 3144 Loss: 0.0445280447602272
Epoch: 3145 Loss: 0.04451160132884979
Epoch: 3146 Loss: 0.044472917914390564
Epoch: 3147 Loss: 0.04446762800216675
Epoch: 3148 Loss: 0.04442683979868889
Epoch: 3149 Loss: 0.044412821531295776
Epoch: 3150 Loss: 0.044381044805049896
Epoch: 3151 Loss: 0.04434046894311905
Epoch: 3152 Loss: 0.04431229457259178
Epoch: 3153 Loss: 0.04431457445025444
Epoch: 3154 Loss: 0.04426492750644684
Epoch: 3155 Loss: 0.044249262660741806
Epoch: 3156 Loss: 0.04421653598546982
Epoch: 3157 Loss: 0.04419359564781189
Epoch: 3158 Loss: 0.04416770488023758
Epoch: 3159 Loss: 0.04415806010365486
Epoch: 3160 Loss: 0.044115159660577774
Epoch: 3161 Loss: 0.044104449450969696
Epoch: 3162 Loss: 0.04407744109630585
Epoch: 3163 Loss: 0.04404310882091522
Epoch: 3164 Loss: 0.04402271658182144
Epoch: 3165 Loss: 0.04398467391729355
Epoch: 3166 Loss: 0.04396635293960571
Epoch: 3167 Loss: 0.043952349573373795
Epoch: 3168 Loss: 0.04390905052423477
Epoch: 3169 Loss: 0.04387935623526573
Epoch: 3170 Loss: 0.043862562626600266
Epoch: 3171 Loss: 0.043844178318977356
Epoch: 3172 Loss: 0.043818484991788864
Epoch: 3173 Loss: 0.0437968485057354
Epoch: 3174 Loss: 0.04378640651702881
Epoch: 3175 Loss: 0.04374246671795845
Epoch: 3176 Loss: 0.043713394552469254
Epoch: 3177 Loss: 0.04370909556746483
Epoch: 3178 Loss: 0.04365173354744911
Epoch: 3179 Loss: 0.04364257678389549
Epoch: 3180 Loss: 0.04361523315310478
Epoch: 3181 Loss: 0.04358368739485741
Epoch: 3182 Loss: 0.043569669127464294
Epoch: 3183 Loss: 0.043555695563554764
Epoch: 3184 Loss: 0.04352181777358055
Epoch: 3185 Loss: 0.04349152371287346
Epoch: 3186 Loss: 0.04347747936844826
Epoch: 3187 Loss: 0.04344789683818817
Epoch: 3188 Loss: 0.0434158630669117
Epoch: 3189 Loss: 0.04338579624891281
Epoch: 3190 Loss: 0.04336143285036087
Epoch: 3191 Loss: 0.043348800390958786
Epoch: 3192 Loss: 0.043318044394254684
Epoch: 3193 Loss: 0.043298620730638504
Epoch: 3194 Loss: 0.0432739332318306
Epoch: 3195 Loss: 0.0432494655251503
Epoch: 3196 Loss: 0.04322017356753349
Epoch: 3197 Loss: 0.04318874329328537
Epoch: 3198 Loss: 0.04317062348127365
Epoch: 3199 Loss: 0.043141867965459824
Epoch: 3200 Loss: 0.04311278089880943
Epoch: 3201 Loss: 0.04309077560901642
Epoch: 3202 Loss: 0.04307098686695099
Epoch: 3203 Loss: 0.043043799698352814
Epoch: 3204 Loss: 0.043017979711294174
Epoch: 3205 Loss: 0.04299019277095795
Epoch: 3206 Loss: 0.04296824336051941
Epoch: 3207 Loss: 0.04295835644006729
Epoch: 3208 Loss: 0.042916491627693176
Epoch: 3209 Loss: 0.04290101304650307
Epoch: 3210 Loss: 0.04288741573691368
Epoch: 3211 Loss: 0.042853470891714096
Epoch: 3212 Loss: 0.04282774776220322
Epoch: 3213 Loss: 0.042809147387742996
Epoch: 3214 Loss: 0.04277324303984642
Epoch: 3215 Loss: 0.04276761785149574
Epoch: 3216 Loss: 0.0427367202937603
Epoch: 3217 Loss: 0.04270971938967705
Epoch: 3218 Loss: 0.04269430413842201
Epoch: 3219 Loss: 0.04266075789928436
Epoch: 3220 Loss: 0.04263601452112198
Epoch: 3221 Loss: 0.04261348396539688
Epoch: 3222 Loss: 0.04258182644844055
Epoch: 3223 Loss: 0.042549241334199905
Epoch: 3224 Loss: 0.04253818094730377
Epoch: 3225 Loss: 0.042508017271757126
Epoch: 3226 Loss: 0.042499423027038574
Epoch: 3227 Loss: 0.04245828837156296
Epoch: 3228 Loss: 0.04244859144091606
Epoch: 3229 Loss: 0.04242004454135895
Epoch: 3230 Loss: 0.0423918291926384
Epoch: 3231 Loss: 0.04235761612653732
Epoch: 3232 Loss: 0.04234565421938896
Epoch: 3233 Loss: 0.04233638942241669
Epoch: 3234 Loss: 0.04230697825551033
Epoch: 3235 Loss: 0.04227150231599808
Epoch: 3236 Loss: 0.0422542430460453
Epoch: 3237 Loss: 0.042226392775774
Epoch: 3238 Loss: 0.04220869392156601
Epoch: 3239 Loss: 0.04218187928199768
Epoch: 3240 Loss: 0.042152710258960724
Epoch: 3241 Loss: 0.042131032794713974
Epoch: 3242 Loss: 0.04213124141097069
Epoch: 3243 Loss: 0.0420796237885952
Epoch: 3244 Loss: 0.04206635430455208
Epoch: 3245 Loss: 0.04204048588871956
Epoch: 3246 Loss: 0.04201767221093178
Epoch: 3247 Loss: 0.041992176324129105
Epoch: 3248 Loss: 0.041980087757110596
Epoch: 3249 Loss: 0.04193843901157379
Epoch: 3250 Loss: 0.04192352667450905
Epoch: 3251 Loss: 0.04190700128674507
Epoch: 3252 Loss: 0.041876111179590225
Epoch: 3253 Loss: 0.04183946177363396
Epoch: 3254 Loss: 0.0418248176574707
Epoch: 3255 Loss: 0.041802097111940384
Epoch: 3256 Loss: 0.04178256168961525
Epoch: 3257 Loss: 0.04174405708909035
Epoch: 3258 Loss: 0.04174480214715004
Epoch: 3259 Loss: 0.04171209782361984
Epoch: 3260 Loss: 0.04170304164290428
Epoch: 3261 Loss: 0.04166588932275772
Epoch: 3262 Loss: 0.04163461551070213
Epoch: 3263 Loss: 0.04160737246274948
Epoch: 3264 Loss: 0.041624195873737335
Epoch: 3265 Loss: 0.04157162085175514
Epoch: 3266 Loss: 0.04154704138636589
Epoch: 3267 Loss: 0.04152558371424675
Epoch: 3268 Loss: 0.041517917066812515
Epoch: 3269 Loss: 0.041477881371974945
Epoch: 3270 Loss: 0.041452087461948395
Epoch: 3271 Loss: 0.041432060301303864
Epoch: 3272 Loss: 0.041417352855205536
Epoch: 3273 Loss: 0.04138197377324104
Epoch: 3274 Loss: 0.04136598855257034
Epoch: 3275 Loss: 0.04133770614862442
Epoch: 3276 Loss: 0.04131888598203659
Epoch: 3277 Loss: 0.04129132628440857
Epoch: 3278 Loss: 0.041285641491413116
Epoch: 3279 Loss: 0.04124163091182709
Epoch: 3280 Loss: 0.04123139753937721
Epoch: 3281 Loss: 0.04120049625635147
Epoch: 3282 Loss: 0.041177041828632355
Epoch: 3283 Loss: 0.04116188362240791
Epoch: 3284 Loss: 0.04114801436662674
Epoch: 3285 Loss: 0.04111577570438385
Epoch: 3286 Loss: 0.041096799075603485
Epoch: 3287 Loss: 0.04107821360230446
Epoch: 3288 Loss: 0.04105633497238159
Epoch: 3289 Loss: 0.04102528467774391
Epoch: 3290 Loss: 0.04099392890930176
Epoch: 3291 Loss: 0.040983591228723526
Epoch: 3292 Loss: 0.04096600040793419
Epoch: 3293 Loss: 0.040937408804893494
Epoch: 3294 Loss: 0.04092376306653023
Epoch: 3295 Loss: 0.040895797312259674
Epoch: 3296 Loss: 0.04085990786552429
Epoch: 3297 Loss: 0.040843140333890915
Epoch: 3298 Loss: 0.04081869497895241
Epoch: 3299 Loss: 0.040798403322696686
Epoch: 3300 Loss: 0.04077821969985962
Epoch: 3301 Loss: 0.040759485214948654
Epoch: 3302 Loss: 0.04073062166571617
Epoch: 3303 Loss: 0.04071618616580963
Epoch: 3304 Loss: 0.04069015383720398
Epoch: 3305 Loss: 0.04066825658082962
Epoch: 3306 Loss: 0.040647510439157486
Epoch: 3307 Loss: 0.04062342271208763
Epoch: 3308 Loss: 0.040599118918180466
Epoch: 3309 Loss: 0.04057256504893303
Epoch: 3310 Loss: 0.04055935889482498
Epoch: 3311 Loss: 0.040530577301979065
Epoch: 3312 Loss: 0.0405132882297039
Epoch: 3313 Loss: 0.040488652884960175
Epoch: 3314 Loss: 0.04046497866511345
Epoch: 3315 Loss: 0.0404505580663681
Epoch: 3316 Loss: 0.040434326976537704
Epoch: 3317 Loss: 0.040398500859737396
Epoch: 3318 Loss: 0.04038742184638977
Epoch: 3319 Loss: 0.040362514555454254
Epoch: 3320 Loss: 0.04035498946905136
Epoch: 3321 Loss: 0.04031210392713547
Epoch: 3322 Loss: 0.04029446095228195
Epoch: 3323 Loss: 0.04027843102812767
Epoch: 3324 Loss: 0.040259815752506256
Epoch: 3325 Loss: 0.04023512825369835
Epoch: 3326 Loss: 0.04020988568663597
Epoch: 3327 Loss: 0.04018598049879074
Epoch: 3328 Loss: 0.04016897454857826
Epoch: 3329 Loss: 0.040139276534318924
Epoch: 3330 Loss: 0.040119122713804245
Epoch: 3331 Loss: 0.040105149149894714
Epoch: 3332 Loss: 0.040071770548820496
Epoch: 3333 Loss: 0.04005390405654907
Epoch: 3334 Loss: 0.04003097489476204
Epoch: 3335 Loss: 0.04002700746059418
Epoch: 3336 Loss: 0.03998878225684166
Epoch: 3337 Loss: 0.0399751253426075
Epoch: 3338 Loss: 0.039959318935871124
Epoch: 3339 Loss: 0.03992517292499542
Epoch: 3340 Loss: 0.03990717604756355
Epoch: 3341 Loss: 0.039887528866529465
Epoch: 3342 Loss: 0.03985961154103279
Epoch: 3343 Loss: 0.039843518286943436
Epoch: 3344 Loss: 0.03980821743607521
Epoch: 3345 Loss: 0.03980807587504387
Epoch: 3346 Loss: 0.03977423533797264
Epoch: 3347 Loss: 0.039749886840581894
Epoch: 3348 Loss: 0.03973535820841789
Epoch: 3349 Loss: 0.039723966270685196
Epoch: 3350 Loss: 0.03969543054699898
Epoch: 3351 Loss: 0.0396728441119194
Epoch: 3352 Loss: 0.0396418534219265
Epoch: 3353 Loss: 0.03963344544172287
Epoch: 3354 Loss: 0.03959759324789047
Epoch: 3355 Loss: 0.03958650678396225
Epoch: 3356 Loss: 0.039560236036777496
Epoch: 3357 Loss: 0.03955374285578728
Epoch: 3358 Loss: 0.03951916843652725
Epoch: 3359 Loss: 0.03951394930481911
Epoch: 3360 Loss: 0.039477620273828506
Epoch: 3361 Loss: 0.039447639137506485
Epoch: 3362 Loss: 0.03944588825106621
Epoch: 3363 Loss: 0.039421986788511276
Epoch: 3364 Loss: 0.03939693421125412
Epoch: 3365 Loss: 0.03937746584415436
Epoch: 3366 Loss: 0.03934656083583832
Epoch: 3367 Loss: 0.03932139277458191
Epoch: 3368 Loss: 0.03932172432541847
Epoch: 3369 Loss: 0.039291221648454666
Epoch: 3370 Loss: 0.03927766531705856
Epoch: 3371 Loss: 0.03924184292554855
Epoch: 3372 Loss: 0.03922506421804428
Epoch: 3373 Loss: 0.03920653089880943
Epoch: 3374 Loss: 0.03918742388486862
Epoch: 3375 Loss: 0.039164330810308456
Epoch: 3376 Loss: 0.03914139047265053
Epoch: 3377 Loss: 0.039121758192777634
Epoch: 3378 Loss: 0.03910404443740845
Epoch: 3379 Loss: 0.03909529745578766
Epoch: 3380 Loss: 0.03905191645026207
Epoch: 3381 Loss: 0.03903908282518387
Epoch: 3382 Loss: 0.039010077714920044
Epoch: 3383 Loss: 0.03901965916156769
Epoch: 3384 Loss: 0.038980044424533844
Epoch: 3385 Loss: 0.03897007927298546
Epoch: 3386 Loss: 0.03892580792307854
Epoch: 3387 Loss: 0.038920436054468155
Epoch: 3388 Loss: 0.038904763758182526
Epoch: 3389 Loss: 0.038879115134477615
Epoch: 3390 Loss: 0.03885047882795334
Epoch: 3391 Loss: 0.03884963318705559
Epoch: 3392 Loss: 0.03881550207734108
Epoch: 3393 Loss: 0.03879927098751068
Epoch: 3394 Loss: 0.03877657651901245
Epoch: 3395 Loss: 0.03875936195254326
Epoch: 3396 Loss: 0.038733478635549545
Epoch: 3397 Loss: 0.03871474787592888
Epoch: 3398 Loss: 0.03869961202144623
Epoch: 3399 Loss: 0.038661595433950424
Epoch: 3400 Loss: 0.03864549100399017
Epoch: 3401 Loss: 0.03863532468676567
Epoch: 3402 Loss: 0.03862340748310089
Epoch: 3403 Loss: 0.03859708085656166
Epoch: 3404 Loss: 0.03856998682022095
Epoch: 3405 Loss: 0.038554493337869644
Epoch: 3406 Loss: 0.03853467479348183
Epoch: 3407 Loss: 0.03850992023944855
Epoch: 3408 Loss: 0.038497813045978546
Epoch: 3409 Loss: 0.0384618304669857
Epoch: 3410 Loss: 0.038450874388217926
Epoch: 3411 Loss: 0.03844086080789566
Epoch: 3412 Loss: 0.03841245174407959
Epoch: 3413 Loss: 0.038390930742025375
Epoch: 3414 Loss: 0.038371700793504715
Epoch: 3415 Loss: 0.03834952041506767
Epoch: 3416 Loss: 0.03833921253681183
Epoch: 3417 Loss: 0.03831332549452782
Epoch: 3418 Loss: 0.03828321397304535
Epoch: 3419 Loss: 0.03827715665102005
Epoch: 3420 Loss: 0.03824686259031296
Epoch: 3421 Loss: 0.038237135857343674
Epoch: 3422 Loss: 0.03820230811834335
Epoch: 3423 Loss: 0.0381917804479599
Epoch: 3424 Loss: 0.03817148879170418
Epoch: 3425 Loss: 0.03815777599811554
Epoch: 3426 Loss: 0.038123469799757004
Epoch: 3427 Loss: 0.03810407221317291
Epoch: 3428 Loss: 0.038083143532276154
Epoch: 3429 Loss: 0.03807144612073898
Epoch: 3430 Loss: 0.038041822612285614
Epoch: 3431 Loss: 0.038034990429878235
Epoch: 3432 Loss: 0.03800101578235626
Epoch: 3433 Loss: 0.03800903260707855
Epoch: 3434 Loss: 0.037969429045915604
Epoch: 3435 Loss: 0.03795768693089485
Epoch: 3436 Loss: 0.037922583520412445
Epoch: 3437 Loss: 0.03790972754359245
Epoch: 3438 Loss: 0.03788768872618675
Epoch: 3439 Loss: 0.03786884993314743
Epoch: 3440 Loss: 0.03784748539328575
Epoch: 3441 Loss: 0.03783447667956352
Epoch: 3442 Loss: 0.03781244903802872
Epoch: 3443 Loss: 0.03778601810336113
Epoch: 3444 Loss: 0.03777061030268669
Epoch: 3445 Loss: 0.037757184356451035
Epoch: 3446 Loss: 0.03772217407822609
Epoch: 3447 Loss: 0.037711553275585175
Epoch: 3448 Loss: 0.037684764713048935
Epoch: 3449 Loss: 0.03767029941082001
Epoch: 3450 Loss: 0.03765079006552696
Epoch: 3451 Loss: 0.03763173148036003
Epoch: 3452 Loss: 0.037619154900312424
Epoch: 3453 Loss: 0.03759661316871643
Epoch: 3454 Loss: 0.03758682683110237
Epoch: 3455 Loss: 0.037560489028692245
Epoch: 3456 Loss: 0.03753812611103058
Epoch: 3457 Loss: 0.03751730918884277
Epoch: 3458 Loss: 0.03750383481383324
Epoch: 3459 Loss: 0.03747982159256935
Epoch: 3460 Loss: 0.03747416287660599
Epoch: 3461 Loss: 0.03744068369269371
Epoch: 3462 Loss: 0.037414517253637314
Epoch: 3463 Loss: 0.03739669919013977
Epoch: 3464 Loss: 0.03737886995077133
Epoch: 3465 Loss: 0.03735789656639099
Epoch: 3466 Loss: 0.037343092262744904
Epoch: 3467 Loss: 0.03732316195964813
Epoch: 3468 Loss: 0.03730364888906479
Epoch: 3469 Loss: 0.03727756068110466
Epoch: 3470 Loss: 0.03727069869637489
Epoch: 3471 Loss: 0.0372481532394886
Epoch: 3472 Loss: 0.0372362919151783
Epoch: 3473 Loss: 0.03720725700259209
Epoch: 3474 Loss: 0.03719431906938553
Epoch: 3475 Loss: 0.03717569261789322
Epoch: 3476 Loss: 0.037158288061618805
Epoch: 3477 Loss: 0.037130873650312424
Epoch: 3478 Loss: 0.03711535036563873
Epoch: 3479 Loss: 0.037092290818691254
Epoch: 3480 Loss: 0.037082821130752563
Epoch: 3481 Loss: 0.03705887869000435
Epoch: 3482 Loss: 0.037040650844573975
Epoch: 3483 Loss: 0.037021562457084656
Epoch: 3484 Loss: 0.03700226917862892
Epoch: 3485 Loss: 0.036987993866205215
Epoch: 3486 Loss: 0.03695513308048248
Epoch: 3487 Loss: 0.03694605827331543
Epoch: 3488 Loss: 0.03692714124917984
Epoch: 3489 Loss: 0.03690784424543381
Epoch: 3490 Loss: 0.03690173104405403
Epoch: 3491 Loss: 0.03687118738889694
Epoch: 3492 Loss: 0.03685050085186958
Epoch: 3493 Loss: 0.036824602633714676
Epoch: 3494 Loss: 0.03681076318025589
Epoch: 3495 Loss: 0.03679722175002098
Epoch: 3496 Loss: 0.03678266331553459
Epoch: 3497 Loss: 0.036762166768312454
Epoch: 3498 Loss: 0.03674367815256119
Epoch: 3499 Loss: 0.03671877086162567
Epoch: 3500 Loss: 0.036704424768686295
Epoch: 3501 Loss: 0.036679770797491074
Epoch: 3502 Loss: 0.03666652739048004
Epoch: 3503 Loss: 0.03664284199476242
Epoch: 3504 Loss: 0.03662990406155586
Epoch: 3505 Loss: 0.03661106899380684
Epoch: 3506 Loss: 0.03658124431967735
Epoch: 3507 Loss: 0.03656342998147011
Epoch: 3508 Loss: 0.03654877468943596
Epoch: 3509 Loss: 0.03652561083436012
Epoch: 3510 Loss: 0.036526743322610855
Epoch: 3511 Loss: 0.03650278598070145
Epoch: 3512 Loss: 0.036485567688941956
Epoch: 3513 Loss: 0.036458443850278854
Epoch: 3514 Loss: 0.036440860480070114
Epoch: 3515 Loss: 0.03642832487821579
Epoch: 3516 Loss: 0.03639887273311615
Epoch: 3517 Loss: 0.0363856665790081
Epoch: 3518 Loss: 0.03636862710118294
Epoch: 3519 Loss: 0.036354582756757736
Epoch: 3520 Loss: 0.036328598856925964
Epoch: 3521 Loss: 0.0363142229616642
Epoch: 3522 Loss: 0.036292240023612976
Epoch: 3523 Loss: 0.036271121352910995
Epoch: 3524 Loss: 0.03626612573862076
Epoch: 3525 Loss: 0.0362384170293808
Epoch: 3526 Loss: 0.036214426159858704
Epoch: 3527 Loss: 0.03620050475001335
Epoch: 3528 Loss: 0.03617307171225548
Epoch: 3529 Loss: 0.03616955131292343
Epoch: 3530 Loss: 0.0361449308693409
Epoch: 3531 Loss: 0.03613085672259331
Epoch: 3532 Loss: 0.03611106798052788
Epoch: 3533 Loss: 0.036099761724472046
Epoch: 3534 Loss: 0.03609021380543709
Epoch: 3535 Loss: 0.03605390340089798
Epoch: 3536 Loss: 0.0360422283411026
Epoch: 3537 Loss: 0.036024678498506546
Epoch: 3538 Loss: 0.03599551320075989
Epoch: 3539 Loss: 0.035982731729745865
Epoch: 3540 Loss: 0.035969849675893784
Epoch: 3541 Loss: 0.035961251705884933
Epoch: 3542 Loss: 0.035935115069150925
Epoch: 3543 Loss: 0.03591189906001091
Epoch: 3544 Loss: 0.035897232592105865
Epoch: 3545 Loss: 0.035873766988515854
Epoch: 3546 Loss: 0.03585868328809738
Epoch: 3547 Loss: 0.03583109751343727
Epoch: 3548 Loss: 0.03583771735429764
Epoch: 3549 Loss: 0.035812631249427795
Epoch: 3550 Loss: 0.03578896448016167
Epoch: 3551 Loss: 0.03577788919210434
Epoch: 3552 Loss: 0.03575413301587105
Epoch: 3553 Loss: 0.03573215380311012
Epoch: 3554 Loss: 0.035706374794244766
Epoch: 3555 Loss: 0.03570623695850372
Epoch: 3556 Loss: 0.03569108992815018
Epoch: 3557 Loss: 0.03566032648086548
Epoch: 3558 Loss: 0.03564184531569481
Epoch: 3559 Loss: 0.035623978823423386
Epoch: 3560 Loss: 0.03561053425073624
Epoch: 3561 Loss: 0.035585615783929825
Epoch: 3562 Loss: 0.03558008745312691
Epoch: 3563 Loss: 0.03555323928594589
Epoch: 3564 Loss: 0.035536088049411774
Epoch: 3565 Loss: 0.03553221747279167
Epoch: 3566 Loss: 0.03550295904278755
Epoch: 3567 Loss: 0.03548318147659302
Epoch: 3568 Loss: 0.03546811640262604
Epoch: 3569 Loss: 0.035440798848867416
Epoch: 3570 Loss: 0.03543087840080261
Epoch: 3571 Loss: 0.0354151614010334
Epoch: 3572 Loss: 0.035401906818151474
Epoch: 3573 Loss: 0.03537510707974434
Epoch: 3574 Loss: 0.03537335619330406
Epoch: 3575 Loss: 0.03534278646111488
Epoch: 3576 Loss: 0.03532733768224716
Epoch: 3577 Loss: 0.035309940576553345
Epoch: 3578 Loss: 0.03528561815619469
Epoch: 3579 Loss: 0.03527068346738815
Epoch: 3580 Loss: 0.03526763245463371
Epoch: 3581 Loss: 0.03523694723844528
Epoch: 3582 Loss: 0.035223692655563354
Epoch: 3583 Loss: 0.03521113842725754
Epoch: 3584 Loss: 0.03517711162567139
Epoch: 3585 Loss: 0.035176124423742294
Epoch: 3586 Loss: 0.03515559434890747
Epoch: 3587 Loss: 0.03513437137007713
Epoch: 3588 Loss: 0.03512568399310112
Epoch: 3589 Loss: 0.03510458767414093
Epoch: 3590 Loss: 0.035084616392850876
Epoch: 3591 Loss: 0.035066477954387665
Epoch: 3592 Loss: 0.03505165874958038
Epoch: 3593 Loss: 0.03503403067588806
Epoch: 3594 Loss: 0.03501865267753601
Epoch: 3595 Loss: 0.03500240296125412
Epoch: 3596 Loss: 0.034984540194272995
Epoch: 3597 Loss: 0.03496435657143593
Epoch: 3598 Loss: 0.03494909033179283
Epoch: 3599 Loss: 0.03492611646652222
Epoch: 3600 Loss: 0.034918300807476044
Epoch: 3601 Loss: 0.034897055476903915
Epoch: 3602 Loss: 0.03487750515341759
Epoch: 3603 Loss: 0.03485901653766632
Epoch: 3604 Loss: 0.0348476804792881
Epoch: 3605 Loss: 0.034828849136829376
Epoch: 3606 Loss: 0.03481549769639969
Epoch: 3607 Loss: 0.03479412570595741
Epoch: 3608 Loss: 0.034773413091897964
Epoch: 3609 Loss: 0.034756921231746674
Epoch: 3610 Loss: 0.034749604761600494
Epoch: 3611 Loss: 0.03472869470715523
Epoch: 3612 Loss: 0.03471408039331436
Epoch: 3613 Loss: 0.03469010442495346
Epoch: 3614 Loss: 0.03467852994799614
Epoch: 3615 Loss: 0.03466327115893364
Epoch: 3616 Loss: 0.03464779257774353
Epoch: 3617 Loss: 0.03462311625480652
Epoch: 3618 Loss: 0.03460446745157242
Epoch: 3619 Loss: 0.034587837755680084
Epoch: 3620 Loss: 0.03457491099834442
Epoch: 3621 Loss: 0.0345589816570282
Epoch: 3622 Loss: 0.03454240411520004
Epoch: 3623 Loss: 0.03454478830099106
Epoch: 3624 Loss: 0.03450565040111542
Epoch: 3625 Loss: 0.034491464495658875
Epoch: 3626 Loss: 0.034474268555641174
Epoch: 3627 Loss: 0.034458182752132416
Epoch: 3628 Loss: 0.03444014489650726
Epoch: 3629 Loss: 0.03442927077412605
Epoch: 3630 Loss: 0.034408070147037506
Epoch: 3631 Loss: 0.03439273685216904
Epoch: 3632 Loss: 0.03438003733754158
Epoch: 3633 Loss: 0.03435787931084633
Epoch: 3634 Loss: 0.03435226157307625
Epoch: 3635 Loss: 0.03431491181254387
Epoch: 3636 Loss: 0.034307144582271576
Epoch: 3637 Loss: 0.03428884968161583
Epoch: 3638 Loss: 0.03427063301205635
Epoch: 3639 Loss: 0.03426271677017212
Epoch: 3640 Loss: 0.03424237295985222
Epoch: 3641 Loss: 0.03422700613737106
Epoch: 3642 Loss: 0.034208714962005615
Epoch: 3643 Loss: 0.03419818729162216
Epoch: 3644 Loss: 0.034176815301179886
Epoch: 3645 Loss: 0.03415043652057648
Epoch: 3646 Loss: 0.03414261341094971
Epoch: 3647 Loss: 0.03412181884050369
Epoch: 3648 Loss: 0.03410591557621956
Epoch: 3649 Loss: 0.03408562019467354
Epoch: 3650 Loss: 0.03407822549343109
Epoch: 3651 Loss: 0.03406304121017456
Epoch: 3652 Loss: 0.03404853492975235
Epoch: 3653 Loss: 0.03402193635702133
Epoch: 3654 Loss: 0.034006666392087936
Epoch: 3655 Loss: 0.03399260714650154
Epoch: 3656 Loss: 0.033986758440732956
Epoch: 3657 Loss: 0.033968109637498856
Epoch: 3658 Loss: 0.03394729644060135
Epoch: 3659 Loss: 0.03393767029047012
Epoch: 3660 Loss: 0.033915925770998
Epoch: 3661 Loss: 0.03389601409435272
Epoch: 3662 Loss: 0.03388134017586708
Epoch: 3663 Loss: 0.033862076699733734
Epoch: 3664 Loss: 0.03385566547513008
Epoch: 3665 Loss: 0.03383738920092583
Epoch: 3666 Loss: 0.03382331505417824
Epoch: 3667 Loss: 0.033794380724430084
Epoch: 3668 Loss: 0.033778514713048935
Epoch: 3669 Loss: 0.03377585485577583
Epoch: 3670 Loss: 0.03374617546796799
Epoch: 3671 Loss: 0.03373762220144272
Epoch: 3672 Loss: 0.03371743857860565
Epoch: 3673 Loss: 0.03369957208633423
Epoch: 3674 Loss: 0.033683743327856064
Epoch: 3675 Loss: 0.03366462513804436
Epoch: 3676 Loss: 0.033655889332294464
Epoch: 3677 Loss: 0.03364041820168495
Epoch: 3678 Loss: 0.033625807613134384
Epoch: 3679 Loss: 0.033608924597501755
Epoch: 3680 Loss: 0.03359527885913849
Epoch: 3681 Loss: 0.03357689827680588
Epoch: 3682 Loss: 0.033559609204530716
Epoch: 3683 Loss: 0.033535219728946686
Epoch: 3684 Loss: 0.03352617472410202
Epoch: 3685 Loss: 0.033511675894260406
Epoch: 3686 Loss: 0.0334942601621151
Epoch: 3687 Loss: 0.03348681330680847
Epoch: 3688 Loss: 0.03347354754805565
Epoch: 3689 Loss: 0.033438507467508316
Epoch: 3690 Loss: 0.033432506024837494
Epoch: 3691 Loss: 0.03341647982597351
Epoch: 3692 Loss: 0.03339892998337746
Epoch: 3693 Loss: 0.0333840474486351
Epoch: 3694 Loss: 0.03337052837014198
Epoch: 3695 Loss: 0.03335835039615631
Epoch: 3696 Loss: 0.03333617001771927
Epoch: 3697 Loss: 0.03332235664129257
Epoch: 3698 Loss: 0.03330601379275322
Epoch: 3699 Loss: 0.03329326584935188
Epoch: 3700 Loss: 0.03327329456806183
Epoch: 3701 Loss: 0.033255431801080704
Epoch: 3702 Loss: 0.03324846923351288
Epoch: 3703 Loss: 0.03322388976812363
Epoch: 3704 Loss: 0.03321395441889763
Epoch: 3705 Loss: 0.03319229185581207
Epoch: 3706 Loss: 0.03317826986312866
Epoch: 3707 Loss: 0.03316468000411987
Epoch: 3708 Loss: 0.033146269619464874
Epoch: 3709 Loss: 0.03313170745968819
Epoch: 3710 Loss: 0.03311290591955185
Epoch: 3711 Loss: 0.03310934081673622
Epoch: 3712 Loss: 0.033091071993112564
Epoch: 3713 Loss: 0.033069100230932236
Epoch: 3714 Loss: 0.033057570457458496
Epoch: 3715 Loss: 0.0330563485622406
Epoch: 3716 Loss: 0.0330270491540432
Epoch: 3717 Loss: 0.03300738334655762
Epoch: 3718 Loss: 0.0329974964261055
Epoch: 3719 Loss: 0.032977111637592316
Epoch: 3720 Loss: 0.03296452388167381
Epoch: 3721 Loss: 0.03294818848371506
Epoch: 3722 Loss: 0.032931044697761536
Epoch: 3723 Loss: 0.032917652279138565
Epoch: 3724 Loss: 0.03289487585425377
Epoch: 3725 Loss: 0.032877322286367416
Epoch: 3726 Loss: 0.032863643020391464
Epoch: 3727 Loss: 0.03286449983716011
Epoch: 3728 Loss: 0.032835640013217926
Epoch: 3729 Loss: 0.03282150998711586
Epoch: 3730 Loss: 0.03281141072511673
Epoch: 3731 Loss: 0.03278585523366928
Epoch: 3732 Loss: 0.03276989981532097
Epoch: 3733 Loss: 0.032759685069322586
Epoch: 3734 Loss: 0.03274683654308319
Epoch: 3735 Loss: 0.032738327980041504
Epoch: 3736 Loss: 0.032717686146497726
Epoch: 3737 Loss: 0.0326974056661129
Epoch: 3738 Loss: 0.032675955444574356
Epoch: 3739 Loss: 0.03267207741737366
Epoch: 3740 Loss: 0.03264883533120155
Epoch: 3741 Loss: 0.032638516277074814
Epoch: 3742 Loss: 0.03262722119688988
Epoch: 3743 Loss: 0.032610319554805756
Epoch: 3744 Loss: 0.03261614590883255
Epoch: 3745 Loss: 0.03257658705115318
Epoch: 3746 Loss: 0.03256003558635712
Epoch: 3747 Loss: 0.03254443779587746
Epoch: 3748 Loss: 0.03253329545259476
Epoch: 3749 Loss: 0.032524723559617996
Epoch: 3750 Loss: 0.03249900043010712
Epoch: 3751 Loss: 0.03249216824769974
Epoch: 3752 Loss: 0.03247211128473282
Epoch: 3753 Loss: 0.0324578657746315
Epoch: 3754 Loss: 0.03243734687566757
Epoch: 3755 Loss: 0.03242839500308037
Epoch: 3756 Loss: 0.03240559995174408
Epoch: 3757 Loss: 0.03239200636744499
Epoch: 3758 Loss: 0.03238799050450325
Epoch: 3759 Loss: 0.03236407786607742
Epoch: 3760 Loss: 0.032347775995731354
Epoch: 3761 Loss: 0.03233662247657776
Epoch: 3762 Loss: 0.0323139913380146
Epoch: 3763 Loss: 0.032308537513017654
Epoch: 3764 Loss: 0.032283637672662735
Epoch: 3765 Loss: 0.03227921947836876
Epoch: 3766 Loss: 0.032255031168460846
Epoch: 3767 Loss: 0.032244499772787094
Epoch: 3768 Loss: 0.03222787380218506
Epoch: 3769 Loss: 0.03220781683921814
Epoch: 3770 Loss: 0.03221278265118599
Epoch: 3771 Loss: 0.032181788235902786
Epoch: 3772 Loss: 0.032164812088012695
Epoch: 3773 Loss: 0.03214704990386963
Epoch: 3774 Loss: 0.032146155834198
Epoch: 3775 Loss: 0.03211730718612671
Epoch: 3776 Loss: 0.032104432582855225
Epoch: 3777 Loss: 0.032092414796352386
Epoch: 3778 Loss: 0.03207624703645706
Epoch: 3779 Loss: 0.03206697851419449
Epoch: 3780 Loss: 0.032047007232904434
Epoch: 3781 Loss: 0.032043445855379105
Epoch: 3782 Loss: 0.03202112019062042
Epoch: 3783 Loss: 0.03200141340494156
Epoch: 3784 Loss: 0.03199147433042526
Epoch: 3785 Loss: 0.03197380155324936
Epoch: 3786 Loss: 0.03196711838245392
Epoch: 3787 Loss: 0.03194350004196167
Epoch: 3788 Loss: 0.03192676603794098
Epoch: 3789 Loss: 0.0319155752658844
Epoch: 3790 Loss: 0.03189924731850624
Epoch: 3791 Loss: 0.03189684450626373
Epoch: 3792 Loss: 0.03185930848121643
Epoch: 3793 Loss: 0.031849995255470276
Epoch: 3794 Loss: 0.03183790296316147
Epoch: 3795 Loss: 0.0318308062851429
Epoch: 3796 Loss: 0.03180922567844391
Epoch: 3797 Loss: 0.03179150074720383
Epoch: 3798 Loss: 0.03178740665316582
Epoch: 3799 Loss: 0.031766850501298904
Epoch: 3800 Loss: 0.03175344318151474
Epoch: 3801 Loss: 0.031742118299007416
Epoch: 3802 Loss: 0.031724411994218826
Epoch: 3803 Loss: 0.03170982003211975
Epoch: 3804 Loss: 0.03168956935405731
Epoch: 3805 Loss: 0.0316854789853096
Epoch: 3806 Loss: 0.03166855499148369
Epoch: 3807 Loss: 0.031647637486457825
Epoch: 3808 Loss: 0.031628936529159546
Epoch: 3809 Loss: 0.0316288135945797
Epoch: 3810 Loss: 0.03160477057099342
Epoch: 3811 Loss: 0.031598664820194244
Epoch: 3812 Loss: 0.03157276660203934
Epoch: 3813 Loss: 0.03156789392232895
Epoch: 3814 Loss: 0.03154296427965164
Epoch: 3815 Loss: 0.0315324142575264
Epoch: 3816 Loss: 0.03151844069361687
Epoch: 3817 Loss: 0.03150629997253418
Epoch: 3818 Loss: 0.03149891272187233
Epoch: 3819 Loss: 0.03147278353571892
Epoch: 3820 Loss: 0.0314694419503212
Epoch: 3821 Loss: 0.03143882006406784
Epoch: 3822 Loss: 0.03143228217959404
Epoch: 3823 Loss: 0.0314197763800621
Epoch: 3824 Loss: 0.03141109645366669
Epoch: 3825 Loss: 0.03139950707554817
Epoch: 3826 Loss: 0.031369730830192566
Epoch: 3827 Loss: 0.03136909380555153
Epoch: 3828 Loss: 0.031342361122369766
Epoch: 3829 Loss: 0.03132734075188637
Epoch: 3830 Loss: 0.031320057809352875
Epoch: 3831 Loss: 0.03130171820521355
Epoch: 3832 Loss: 0.03129176050424576
Epoch: 3833 Loss: 0.031265739351511
Epoch: 3834 Loss: 0.03126020357012749
Epoch: 3835 Loss: 0.031244849786162376
Epoch: 3836 Loss: 0.03124346025288105
Epoch: 3837 Loss: 0.03121544048190117
Epoch: 3838 Loss: 0.031200289726257324
Epoch: 3839 Loss: 0.031199300661683083
Epoch: 3840 Loss: 0.031190956011414528
Epoch: 3841 Loss: 0.031162353232502937
Epoch: 3842 Loss: 0.031153645366430283
Epoch: 3843 Loss: 0.03112851083278656
Epoch: 3844 Loss: 0.031126657500863075
Epoch: 3845 Loss: 0.03109964169561863
Epoch: 3846 Loss: 0.031086834147572517
Epoch: 3847 Loss: 0.03107902593910694
Epoch: 3848 Loss: 0.031056636944413185
Epoch: 3849 Loss: 0.031052161008119583
Epoch: 3850 Loss: 0.03102877363562584
Epoch: 3851 Loss: 0.031021056696772575
Epoch: 3852 Loss: 0.0310222078114748
Epoch: 3853 Loss: 0.030991801992058754
Epoch: 3854 Loss: 0.030980367213487625
Epoch: 3855 Loss: 0.030969755724072456
Epoch: 3856 Loss: 0.03095032088458538
Epoch: 3857 Loss: 0.03093492053449154
Epoch: 3858 Loss: 0.03092159330844879
Epoch: 3859 Loss: 0.030905772000551224
Epoch: 3860 Loss: 0.030895689502358437
Epoch: 3861 Loss: 0.030889583751559258
Epoch: 3862 Loss: 0.0308647770434618
Epoch: 3863 Loss: 0.030850308015942574
Epoch: 3864 Loss: 0.030837705358862877
Epoch: 3865 Loss: 0.030818413943052292
Epoch: 3866 Loss: 0.030822791159152985
Epoch: 3867 Loss: 0.030799314379692078
Epoch: 3868 Loss: 0.030788175761699677
Epoch: 3869 Loss: 0.030767466872930527
Epoch: 3870 Loss: 0.03075231984257698
Epoch: 3871 Loss: 0.030747048556804657
Epoch: 3872 Loss: 0.030724039301276207
Epoch: 3873 Loss: 0.03071114979684353
Epoch: 3874 Loss: 0.030712679028511047
Epoch: 3875 Loss: 0.03068516217172146
Epoch: 3876 Loss: 0.03066800907254219
Epoch: 3877 Loss: 0.030657809227705002
Epoch: 3878 Loss: 0.030651839450001717
Epoch: 3879 Loss: 0.030634893104434013
Epoch: 3880 Loss: 0.0306223314255476
Epoch: 3881 Loss: 0.030606962740421295
Epoch: 3882 Loss: 0.030590087175369263
Epoch: 3883 Loss: 0.030584273859858513
Epoch: 3884 Loss: 0.030565455555915833
Epoch: 3885 Loss: 0.030545677989721298
Epoch: 3886 Loss: 0.030538806691765785
Epoch: 3887 Loss: 0.030529795214533806
Epoch: 3888 Loss: 0.030514612793922424
Epoch: 3889 Loss: 0.03049917332828045
Epoch: 3890 Loss: 0.030477549880743027
Epoch: 3891 Loss: 0.030468130484223366
Epoch: 3892 Loss: 0.03044959530234337
Epoch: 3893 Loss: 0.030438335612416267
Epoch: 3894 Loss: 0.030423544347286224
Epoch: 3895 Loss: 0.030413450673222542
Epoch: 3896 Loss: 0.030401770025491714
Epoch: 3897 Loss: 0.030393144115805626
Epoch: 3898 Loss: 0.03036554530262947
Epoch: 3899 Loss: 0.030360229313373566
Epoch: 3900 Loss: 0.030345838516950607
Epoch: 3901 Loss: 0.030330467969179153
Epoch: 3902 Loss: 0.030316388234496117
Epoch: 3903 Loss: 0.030306294560432434
Epoch: 3904 Loss: 0.030295826494693756
Epoch: 3905 Loss: 0.0302814282476902
Epoch: 3906 Loss: 0.030259588733315468
Epoch: 3907 Loss: 0.030247751623392105
Epoch: 3908 Loss: 0.030238237231969833
Epoch: 3909 Loss: 0.03022272139787674
Epoch: 3910 Loss: 0.030215105041861534
Epoch: 3911 Loss: 0.030202167108654976
Epoch: 3912 Loss: 0.030191248282790184
Epoch: 3913 Loss: 0.030174020677804947
Epoch: 3914 Loss: 0.030164025723934174
Epoch: 3915 Loss: 0.030143801122903824
Epoch: 3916 Loss: 0.030127564445137978
Epoch: 3917 Loss: 0.030118802562355995
Epoch: 3918 Loss: 0.030111173167824745
Epoch: 3919 Loss: 0.030090846121311188
Epoch: 3920 Loss: 0.030088504776358604
Epoch: 3921 Loss: 0.03006785362958908
Epoch: 3922 Loss: 0.030047116801142693
Epoch: 3923 Loss: 0.030039967969059944
Epoch: 3924 Loss: 0.030025644227862358
Epoch: 3925 Loss: 0.03002004884183407
Epoch: 3926 Loss: 0.030004803091287613
Epoch: 3927 Loss: 0.02998848631978035
Epoch: 3928 Loss: 0.02997235581278801
Epoch: 3929 Loss: 0.029959099367260933
Epoch: 3930 Loss: 0.029943065717816353
Epoch: 3931 Loss: 0.02993442304432392
Epoch: 3932 Loss: 0.02991921454668045
Epoch: 3933 Loss: 0.029909009113907814
Epoch: 3934 Loss: 0.029889384284615517
Epoch: 3935 Loss: 0.02987772598862648
Epoch: 3936 Loss: 0.029860224574804306
Epoch: 3937 Loss: 0.029845723882317543
Epoch: 3938 Loss: 0.02984773740172386
Epoch: 3939 Loss: 0.029825828969478607
Epoch: 3940 Loss: 0.029826998710632324
Epoch: 3941 Loss: 0.029803097248077393
Epoch: 3942 Loss: 0.029787689447402954
Epoch: 3943 Loss: 0.029780957847833633
Epoch: 3944 Loss: 0.02976185828447342
Epoch: 3945 Loss: 0.02974610961973667
Epoch: 3946 Loss: 0.029739031568169594
Epoch: 3947 Loss: 0.029742557555437088
Epoch: 3948 Loss: 0.029708102345466614
Epoch: 3949 Loss: 0.029697002843022346
Epoch: 3950 Loss: 0.029684003442525864
Epoch: 3951 Loss: 0.02967100217938423
Epoch: 3952 Loss: 0.029662448912858963
Epoch: 3953 Loss: 0.029640361666679382
Epoch: 3954 Loss: 0.029633736237883568
Epoch: 3955 Loss: 0.02962440811097622
Epoch: 3956 Loss: 0.02960140071809292
Epoch: 3957 Loss: 0.029600683599710464
Epoch: 3958 Loss: 0.02957839146256447
Epoch: 3959 Loss: 0.029569512233138084
Epoch: 3960 Loss: 0.029553048312664032
Epoch: 3961 Loss: 0.029539205133914948
Epoch: 3962 Loss: 0.029532290995121002
Epoch: 3963 Loss: 0.029514091089367867
Epoch: 3964 Loss: 0.029503701254725456
Epoch: 3965 Loss: 0.02948535420000553
Epoch: 3966 Loss: 0.02947210520505905
Epoch: 3967 Loss: 0.029454614967107773
Epoch: 3968 Loss: 0.029456665739417076
Epoch: 3969 Loss: 0.029445664957165718
Epoch: 3970 Loss: 0.029431208968162537
Epoch: 3971 Loss: 0.029415611177682877
Epoch: 3972 Loss: 0.029395906254649162
Epoch: 3973 Loss: 0.029399193823337555
Epoch: 3974 Loss: 0.02937118336558342
Epoch: 3975 Loss: 0.029363393783569336
Epoch: 3976 Loss: 0.02935890294611454
Epoch: 3977 Loss: 0.029340175911784172
Epoch: 3978 Loss: 0.02932199090719223
Epoch: 3979 Loss: 0.02931436151266098
Epoch: 3980 Loss: 0.02929387427866459
Epoch: 3981 Loss: 0.029283257201313972
Epoch: 3982 Loss: 0.02927333116531372
Epoch: 3983 Loss: 0.029261359944939613
Epoch: 3984 Loss: 0.029247604310512543
Epoch: 3985 Loss: 0.029230397194623947
Epoch: 3986 Loss: 0.029229359701275826
Epoch: 3987 Loss: 0.029208332300186157
Epoch: 3988 Loss: 0.029199469834566116
Epoch: 3989 Loss: 0.029183438047766685
Epoch: 3990 Loss: 0.029186563566327095
Epoch: 3991 Loss: 0.029155075550079346
Epoch: 3992 Loss: 0.02914436347782612
Epoch: 3993 Loss: 0.02912985347211361
Epoch: 3994 Loss: 0.029119741171598434
Epoch: 3995 Loss: 0.029115229845046997
Epoch: 3996 Loss: 0.029093598946928978
Epoch: 3997 Loss: 0.0290937889367342
Epoch: 3998 Loss: 0.029068544507026672
Epoch: 3999 Loss: 0.029053224250674248
Epoch: 4000 Loss: 0.02904629148542881
Epoch: 4001 Loss: 0.02903527207672596
Epoch: 4002 Loss: 0.02901754155755043
Epoch: 4003 Loss: 0.029013242572546005
Epoch: 4004 Loss: 0.028990617021918297
Epoch: 4005 Loss: 0.028988823294639587
Epoch: 4006 Loss: 0.028970753774046898
Epoch: 4007 Loss: 0.028950927779078484
Epoch: 4008 Loss: 0.028945373371243477
Epoch: 4009 Loss: 0.028932122513651848
Epoch: 4010 Loss: 0.02891920693218708
Epoch: 4011 Loss: 0.028920801356434822
Epoch: 4012 Loss: 0.028899073600769043
Epoch: 4013 Loss: 0.02887985296547413
Epoch: 4014 Loss: 0.02886800654232502
Epoch: 4015 Loss: 0.02885720692574978
Epoch: 4016 Loss: 0.028849689289927483
Epoch: 4017 Loss: 0.028832485899329185
Epoch: 4018 Loss: 0.028821753337979317
Epoch: 4019 Loss: 0.028803063556551933
Epoch: 4020 Loss: 0.028805265203118324
Epoch: 4021 Loss: 0.028787920251488686
Epoch: 4022 Loss: 0.02877049148082733
Epoch: 4023 Loss: 0.028766674920916557
Epoch: 4024 Loss: 0.028747932985424995
Epoch: 4025 Loss: 0.028735170140862465
Epoch: 4026 Loss: 0.028725994750857353
Epoch: 4027 Loss: 0.028712598606944084
Epoch: 4028 Loss: 0.028699534013867378
Epoch: 4029 Loss: 0.02868821658194065
Epoch: 4030 Loss: 0.028667282313108444
Epoch: 4031 Loss: 0.028657246381044388
Epoch: 4032 Loss: 0.028651876375079155
Epoch: 4033 Loss: 0.028639765456318855
Epoch: 4034 Loss: 0.02863326296210289
Epoch: 4035 Loss: 0.028611784800887108
Epoch: 4036 Loss: 0.028606900945305824
Epoch: 4037 Loss: 0.02858678624033928
Epoch: 4038 Loss: 0.0285757165402174
Epoch: 4039 Loss: 0.028564903885126114
Epoch: 4040 Loss: 0.028556441888213158
Epoch: 4041 Loss: 0.028539810329675674
Epoch: 4042 Loss: 0.0285227932035923
Epoch: 4043 Loss: 0.028512654826045036
Epoch: 4044 Loss: 0.02850465662777424
Epoch: 4045 Loss: 0.028485847637057304
Epoch: 4046 Loss: 0.028478171676397324
Epoch: 4047 Loss: 0.028461797162890434
Epoch: 4048 Loss: 0.028460055589675903
Epoch: 4049 Loss: 0.028438197448849678
Epoch: 4050 Loss: 0.028426749631762505
Epoch: 4051 Loss: 0.0284211877733469
Epoch: 4052 Loss: 0.028401901945471764
Epoch: 4053 Loss: 0.02839026041328907
Epoch: 4054 Loss: 0.0283745676279068
Epoch: 4055 Loss: 0.0283685140311718
Epoch: 4056 Loss: 0.02835613116621971
Epoch: 4057 Loss: 0.028350071981549263
Epoch: 4058 Loss: 0.028335219249129295
Epoch: 4059 Loss: 0.02832503244280815
Epoch: 4060 Loss: 0.028317563235759735
Epoch: 4061 Loss: 0.02829981967806816
Epoch: 4062 Loss: 0.028283869847655296
Epoch: 4063 Loss: 0.02826528437435627
Epoch: 4064 Loss: 0.02826179750263691
Epoch: 4065 Loss: 0.02826019749045372
Epoch: 4066 Loss: 0.028233937919139862
Epoch: 4067 Loss: 0.028220387175679207
Epoch: 4068 Loss: 0.028215236961841583
Epoch: 4069 Loss: 0.02820425294339657
Epoch: 4070 Loss: 0.02818210981786251
Epoch: 4071 Loss: 0.028181826695799828
Epoch: 4072 Loss: 0.028166210278868675
Epoch: 4073 Loss: 0.028152387589216232
Epoch: 4074 Loss: 0.028141481801867485
Epoch: 4075 Loss: 0.028127368539571762
Epoch: 4076 Loss: 0.02812209725379944
Epoch: 4077 Loss: 0.028104402124881744
Epoch: 4078 Loss: 0.028095312416553497
Epoch: 4079 Loss: 0.02807801589369774
Epoch: 4080 Loss: 0.02807694673538208
Epoch: 4081 Loss: 0.02806241810321808
Epoch: 4082 Loss: 0.028043074533343315
Epoch: 4083 Loss: 0.02803579717874527
Epoch: 4084 Loss: 0.02802303060889244
Epoch: 4085 Loss: 0.028013812378048897
Epoch: 4086 Loss: 0.028004303574562073
Epoch: 4087 Loss: 0.027987031266093254
Epoch: 4088 Loss: 0.02797612175345421
Epoch: 4089 Loss: 0.02795897237956524
Epoch: 4090 Loss: 0.02795650064945221
Epoch: 4091 Loss: 0.027936600148677826
Epoch: 4092 Loss: 0.02793440967798233
Epoch: 4093 Loss: 0.02792220376431942
Epoch: 4094 Loss: 0.027909766882658005
Epoch: 4095 Loss: 0.02788788080215454
Epoch: 4096 Loss: 0.027884064242243767
Epoch: 4097 Loss: 0.027866968885064125
Epoch: 4098 Loss: 0.027853382751345634
Epoch: 4099 Loss: 0.027849409729242325
Epoch: 4100 Loss: 0.027831248939037323
Epoch: 4101 Loss: 0.027818014845252037
Epoch: 4102 Loss: 0.027813170105218887
Epoch: 4103 Loss: 0.027811594307422638
Epoch: 4104 Loss: 0.02779388427734375
Epoch: 4105 Loss: 0.027772104367613792
Epoch: 4106 Loss: 0.02776312828063965
Epoch: 4107 Loss: 0.02775532938539982
Epoch: 4108 Loss: 0.02773929201066494
Epoch: 4109 Loss: 0.027729948982596397
Epoch: 4110 Loss: 0.027714267373085022
Epoch: 4111 Loss: 0.027706852182745934
Epoch: 4112 Loss: 0.027690552175045013
Epoch: 4113 Loss: 0.0276794396340847
Epoch: 4114 Loss: 0.027676105499267578
Epoch: 4115 Loss: 0.0276565570384264
Epoch: 4116 Loss: 0.027655744925141335
Epoch: 4117 Loss: 0.027642162516713142
Epoch: 4118 Loss: 0.027629293501377106
Epoch: 4119 Loss: 0.02761797048151493
Epoch: 4120 Loss: 0.027607088908553123
Epoch: 4121 Loss: 0.027591237798333168
Epoch: 4122 Loss: 0.027579184621572495
Epoch: 4123 Loss: 0.02756970189511776
Epoch: 4124 Loss: 0.02754869870841503
Epoch: 4125 Loss: 0.027542423456907272
Epoch: 4126 Loss: 0.027533726766705513
Epoch: 4127 Loss: 0.027525177225470543
Epoch: 4128 Loss: 0.027511300519108772
Epoch: 4129 Loss: 0.027492228895425797
Epoch: 4130 Loss: 0.027486838400363922
Epoch: 4131 Loss: 0.02748681791126728
Epoch: 4132 Loss: 0.02745814248919487
Epoch: 4133 Loss: 0.0274615790694952
Epoch: 4134 Loss: 0.02744222991168499
Epoch: 4135 Loss: 0.02742813341319561
Epoch: 4136 Loss: 0.027418427169322968
Epoch: 4137 Loss: 0.027405137196183205
Epoch: 4138 Loss: 0.027404651045799255
Epoch: 4139 Loss: 0.027389075607061386
Epoch: 4140 Loss: 0.027373913675546646
Epoch: 4141 Loss: 0.027368852868676186
Epoch: 4142 Loss: 0.02735232003033161
Epoch: 4143 Loss: 0.027345266193151474
Epoch: 4144 Loss: 0.02734128199517727
Epoch: 4145 Loss: 0.027315646409988403
Epoch: 4146 Loss: 0.0273029375821352
Epoch: 4147 Loss: 0.027298657223582268
Epoch: 4148 Loss: 0.027283530682325363
Epoch: 4149 Loss: 0.027274902909994125
Epoch: 4150 Loss: 0.02725650556385517
Epoch: 4151 Loss: 0.027242762967944145
Epoch: 4152 Loss: 0.02723987214267254
Epoch: 4153 Loss: 0.027228740975260735
Epoch: 4154 Loss: 0.02721780352294445
Epoch: 4155 Loss: 0.027203815057873726
Epoch: 4156 Loss: 0.027204131707549095
Epoch: 4157 Loss: 0.027182849124073982
Epoch: 4158 Loss: 0.02716837264597416
Epoch: 4159 Loss: 0.027162831276655197
Epoch: 4160 Loss: 0.02715083584189415
Epoch: 4161 Loss: 0.027141928672790527
Epoch: 4162 Loss: 0.027121301740407944
Epoch: 4163 Loss: 0.027122581377625465
Epoch: 4164 Loss: 0.027105819433927536
Epoch: 4165 Loss: 0.027095217257738113
Epoch: 4166 Loss: 0.027083177119493484
Epoch: 4167 Loss: 0.027067333459854126
Epoch: 4168 Loss: 0.027058886364102364
Epoch: 4169 Loss: 0.027051445096731186
Epoch: 4170 Loss: 0.02703983522951603
Epoch: 4171 Loss: 0.027027221396565437
Epoch: 4172 Loss: 0.02702844701707363
Epoch: 4173 Loss: 0.027002742514014244
Epoch: 4174 Loss: 0.027000557631254196
Epoch: 4175 Loss: 0.0269849244505167
Epoch: 4176 Loss: 0.02696819044649601
Epoch: 4177 Loss: 0.02696124278008938
Epoch: 4178 Loss: 0.026942178606987
Epoch: 4179 Loss: 0.026950778439641
Epoch: 4180 Loss: 0.02692522294819355
Epoch: 4181 Loss: 0.026912568137049675
Epoch: 4182 Loss: 0.026911642402410507
Epoch: 4183 Loss: 0.02689531445503235
Epoch: 4184 Loss: 0.026882562786340714
Epoch: 4185 Loss: 0.02687131240963936
Epoch: 4186 Loss: 0.0268595889210701
Epoch: 4187 Loss: 0.02685561776161194
Epoch: 4188 Loss: 0.026837440207600594
Epoch: 4189 Loss: 0.026823189109563828
Epoch: 4190 Loss: 0.026810957118868828
Epoch: 4191 Loss: 0.026808103546500206
Epoch: 4192 Loss: 0.02678736299276352
Epoch: 4193 Loss: 0.02678847871720791
Epoch: 4194 Loss: 0.026773981750011444
Epoch: 4195 Loss: 0.02676323987543583
Epoch: 4196 Loss: 0.026752345263957977
Epoch: 4197 Loss: 0.026738718152046204
Epoch: 4198 Loss: 0.026727400720119476
Epoch: 4199 Loss: 0.026715358719229698
Epoch: 4200 Loss: 0.026709692552685738
Epoch: 4201 Loss: 0.026695024222135544
Epoch: 4202 Loss: 0.026692915707826614
Epoch: 4203 Loss: 0.02666868455708027
Epoch: 4204 Loss: 0.026667611673474312
Epoch: 4205 Loss: 0.026656219735741615
Epoch: 4206 Loss: 0.02664279006421566
Epoch: 4207 Loss: 0.02662592940032482
Epoch: 4208 Loss: 0.0266161747276783
Epoch: 4209 Loss: 0.02660406194627285
Epoch: 4210 Loss: 0.02659931592643261
Epoch: 4211 Loss: 0.02659260667860508
Epoch: 4212 Loss: 0.026580732315778732
Epoch: 4213 Loss: 0.026561541482806206
Epoch: 4214 Loss: 0.026546763256192207
Epoch: 4215 Loss: 0.0265392754226923
Epoch: 4216 Loss: 0.02653447538614273
Epoch: 4217 Loss: 0.026520853862166405
Epoch: 4218 Loss: 0.02650580182671547
Epoch: 4219 Loss: 0.02648923546075821
Epoch: 4220 Loss: 0.02648695930838585
Epoch: 4221 Loss: 0.026470784097909927
Epoch: 4222 Loss: 0.026464281603693962
Epoch: 4223 Loss: 0.0264506246894598
Epoch: 4224 Loss: 0.02644680254161358
Epoch: 4225 Loss: 0.026422932744026184
Epoch: 4226 Loss: 0.026415526866912842
Epoch: 4227 Loss: 0.0264009777456522
Epoch: 4228 Loss: 0.026394948363304138
Epoch: 4229 Loss: 0.02637876197695732
Epoch: 4230 Loss: 0.026371031999588013
Epoch: 4231 Loss: 0.026355717331171036
Epoch: 4232 Loss: 0.026351777836680412
Epoch: 4233 Loss: 0.026337429881095886
Epoch: 4234 Loss: 0.02632524073123932
Epoch: 4235 Loss: 0.026314781978726387
Epoch: 4236 Loss: 0.0263004619628191
Epoch: 4237 Loss: 0.02629060484468937
Epoch: 4238 Loss: 0.026282144710421562
Epoch: 4239 Loss: 0.026279916986823082
Epoch: 4240 Loss: 0.026259152218699455
Epoch: 4241 Loss: 0.026243656873703003
Epoch: 4242 Loss: 0.02624192088842392
Epoch: 4243 Loss: 0.02622334100306034
Epoch: 4244 Loss: 0.02621658891439438
Epoch: 4245 Loss: 0.026208186522126198
Epoch: 4246 Loss: 0.026198623701930046
Epoch: 4247 Loss: 0.026174714788794518
Epoch: 4248 Loss: 0.026177696883678436
Epoch: 4249 Loss: 0.026157140731811523
Epoch: 4250 Loss: 0.02614542655646801
Epoch: 4251 Loss: 0.02614154852926731
Epoch: 4252 Loss: 0.026122473180294037
Epoch: 4253 Loss: 0.026117373257875443
Epoch: 4254 Loss: 0.026100341230630875
Epoch: 4255 Loss: 0.02610091120004654
Epoch: 4256 Loss: 0.02608049288392067
Epoch: 4257 Loss: 0.026065263897180557
Epoch: 4258 Loss: 0.026067396625876427
Epoch: 4259 Loss: 0.026051117107272148
Epoch: 4260 Loss: 0.0260382778942585
Epoch: 4261 Loss: 0.026028893887996674
Epoch: 4262 Loss: 0.026020964607596397
Epoch: 4263 Loss: 0.026004977524280548
Epoch: 4264 Loss: 0.025994544848799706
Epoch: 4265 Loss: 0.025989582762122154
Epoch: 4266 Loss: 0.025970464572310448
Epoch: 4267 Loss: 0.025968583300709724
Epoch: 4268 Loss: 0.025948774069547653
Epoch: 4269 Loss: 0.025944631546735764
Epoch: 4270 Loss: 0.025927314534783363
Epoch: 4271 Loss: 0.025922128930687904
Epoch: 4272 Loss: 0.02590966410934925
Epoch: 4273 Loss: 0.02589971385896206
Epoch: 4274 Loss: 0.025891438126564026
Epoch: 4275 Loss: 0.025878874585032463
Epoch: 4276 Loss: 0.025868473574519157
Epoch: 4277 Loss: 0.02586633339524269
Epoch: 4278 Loss: 0.02584097906947136
Epoch: 4279 Loss: 0.0258319228887558
Epoch: 4280 Loss: 0.025825079530477524
Epoch: 4281 Loss: 0.025812748819589615
Epoch: 4282 Loss: 0.025798238813877106
Epoch: 4283 Loss: 0.025794919580221176
Epoch: 4284 Loss: 0.02577805705368519
Epoch: 4285 Loss: 0.025773786008358
Epoch: 4286 Loss: 0.025761010125279427
Epoch: 4287 Loss: 0.02574748918414116
Epoch: 4288 Loss: 0.025745699182152748
Epoch: 4289 Loss: 0.0257257092744112
Epoch: 4290 Loss: 0.025716198608279228
Epoch: 4291 Loss: 0.02570563368499279
Epoch: 4292 Loss: 0.025693539530038834
Epoch: 4293 Loss: 0.025687506422400475
Epoch: 4294 Loss: 0.025679217651486397
Epoch: 4295 Loss: 0.025659946724772453
Epoch: 4296 Loss: 0.02565251663327217
Epoch: 4297 Loss: 0.02564547210931778
Epoch: 4298 Loss: 0.02563493698835373
Epoch: 4299 Loss: 0.025625115260481834
Epoch: 4300 Loss: 0.02560316026210785
Epoch: 4301 Loss: 0.02560393512248993
Epoch: 4302 Loss: 0.02558961883187294
Epoch: 4303 Loss: 0.02558361180126667
Epoch: 4304 Loss: 0.02557268552482128
Epoch: 4305 Loss: 0.02555776759982109
Epoch: 4306 Loss: 0.025549357756972313
Epoch: 4307 Loss: 0.025539904832839966
Epoch: 4308 Loss: 0.025526348501443863
Epoch: 4309 Loss: 0.025516297668218613
Epoch: 4310 Loss: 0.025504356250166893
Epoch: 4311 Loss: 0.025505945086479187
Epoch: 4312 Loss: 0.025484107434749603
Epoch: 4313 Loss: 0.025478309020400047
Epoch: 4314 Loss: 0.02546403370797634
Epoch: 4315 Loss: 0.025451453402638435
Epoch: 4316 Loss: 0.025445586070418358
Epoch: 4317 Loss: 0.025432145223021507
Epoch: 4318 Loss: 0.025416933000087738
Epoch: 4319 Loss: 0.025416169315576553
Epoch: 4320 Loss: 0.025404734537005424
Epoch: 4321 Loss: 0.025388166308403015
Epoch: 4322 Loss: 0.025383111089468002
Epoch: 4323 Loss: 0.025372300297021866
Epoch: 4324 Loss: 0.025367707014083862
Epoch: 4325 Loss: 0.02535013109445572
Epoch: 4326 Loss: 0.025343796238303185
Epoch: 4327 Loss: 0.02532741241157055
Epoch: 4328 Loss: 0.02532246708869934
Epoch: 4329 Loss: 0.02531382255256176
Epoch: 4330 Loss: 0.025298207998275757
Epoch: 4331 Loss: 0.025292763486504555
Epoch: 4332 Loss: 0.02527843974530697
Epoch: 4333 Loss: 0.025270821526646614
Epoch: 4334 Loss: 0.02525770664215088
Epoch: 4335 Loss: 0.025249041616916656
Epoch: 4336 Loss: 0.02523805946111679
Epoch: 4337 Loss: 0.025228891521692276
Epoch: 4338 Loss: 0.025219859555363655
Epoch: 4339 Loss: 0.02520795352756977
Epoch: 4340 Loss: 0.025197766721248627
Epoch: 4341 Loss: 0.025185374543070793
Epoch: 4342 Loss: 0.025172244757413864
Epoch: 4343 Loss: 0.025168217718601227
Epoch: 4344 Loss: 0.025150293484330177
Epoch: 4345 Loss: 0.0251497533172369
Epoch: 4346 Loss: 0.025134854018688202
Epoch: 4347 Loss: 0.025126762688159943
Epoch: 4348 Loss: 0.025117909535765648
Epoch: 4349 Loss: 0.025106362998485565
Epoch: 4350 Loss: 0.025095395743846893
Epoch: 4351 Loss: 0.025088785216212273
Epoch: 4352 Loss: 0.025078460574150085
Epoch: 4353 Loss: 0.025065509602427483
Epoch: 4354 Loss: 0.02506418339908123
Epoch: 4355 Loss: 0.025043601170182228
Epoch: 4356 Loss: 0.025036361068487167
Epoch: 4357 Loss: 0.02502872608602047
Epoch: 4358 Loss: 0.025012673810124397
Epoch: 4359 Loss: 0.02500639669597149
Epoch: 4360 Loss: 0.025000547990202904
Epoch: 4361 Loss: 0.024984899908304214
Epoch: 4362 Loss: 0.024978619068861008
Epoch: 4363 Loss: 0.024964308366179466
Epoch: 4364 Loss: 0.024957550689578056
Epoch: 4365 Loss: 0.024945074692368507
Epoch: 4366 Loss: 0.024935049936175346
Epoch: 4367 Loss: 0.02492157556116581
Epoch: 4368 Loss: 0.02491793967783451
Epoch: 4369 Loss: 0.024905648082494736
Epoch: 4370 Loss: 0.024893583729863167
Epoch: 4371 Loss: 0.02488333359360695
Epoch: 4372 Loss: 0.024877849966287613
Epoch: 4373 Loss: 0.02486603334546089
Epoch: 4374 Loss: 0.02486570179462433
Epoch: 4375 Loss: 0.024847259745001793
Epoch: 4376 Loss: 0.02483959309756756
Epoch: 4377 Loss: 0.024823946878314018
Epoch: 4378 Loss: 0.024826256558299065
Epoch: 4379 Loss: 0.024808015674352646
Epoch: 4380 Loss: 0.02479933574795723
Epoch: 4381 Loss: 0.024788083508610725
Epoch: 4382 Loss: 0.02477995865046978
Epoch: 4383 Loss: 0.02476959303021431
Epoch: 4384 Loss: 0.024753646925091743
Epoch: 4385 Loss: 0.024746349081397057
Epoch: 4386 Loss: 0.024735335260629654
Epoch: 4387 Loss: 0.024730056524276733
Epoch: 4388 Loss: 0.024720240384340286
Epoch: 4389 Loss: 0.024714991450309753
Epoch: 4390 Loss: 0.024702807888388634
Epoch: 4391 Loss: 0.024688689038157463
Epoch: 4392 Loss: 0.02467636577785015
Epoch: 4393 Loss: 0.024668382480740547
Epoch: 4394 Loss: 0.024663930758833885
Epoch: 4395 Loss: 0.024657631292939186
Epoch: 4396 Loss: 0.02464243955910206
Epoch: 4397 Loss: 0.02462981827557087
Epoch: 4398 Loss: 0.02462652511894703
Epoch: 4399 Loss: 0.024612724781036377
Epoch: 4400 Loss: 0.024600088596343994
Epoch: 4401 Loss: 0.024596519768238068
Epoch: 4402 Loss: 0.024584852159023285
Epoch: 4403 Loss: 0.024571487680077553
Epoch: 4404 Loss: 0.024565858766436577
Epoch: 4405 Loss: 0.024549366906285286
Epoch: 4406 Loss: 0.024541830644011497
Epoch: 4407 Loss: 0.02453457936644554
Epoch: 4408 Loss: 0.024527618661522865
Epoch: 4409 Loss: 0.024514643475413322
Epoch: 4410 Loss: 0.024504972621798515
Epoch: 4411 Loss: 0.024496998637914658
Epoch: 4412 Loss: 0.024484992027282715
Epoch: 4413 Loss: 0.024477185681462288
Epoch: 4414 Loss: 0.02446906641125679
Epoch: 4415 Loss: 0.024459686130285263
Epoch: 4416 Loss: 0.024450816214084625
Epoch: 4417 Loss: 0.024439550936222076
Epoch: 4418 Loss: 0.024436064064502716
Epoch: 4419 Loss: 0.024417607113718987
Epoch: 4420 Loss: 0.024410167708992958
Epoch: 4421 Loss: 0.024395577609539032
Epoch: 4422 Loss: 0.02439233846962452
Epoch: 4423 Loss: 0.02437680773437023
Epoch: 4424 Loss: 0.024368375539779663
Epoch: 4425 Loss: 0.02436058409512043
Epoch: 4426 Loss: 0.024359239265322685
Epoch: 4427 Loss: 0.02434307150542736
Epoch: 4428 Loss: 0.024331239983439445
Epoch: 4429 Loss: 0.02432885766029358
Epoch: 4430 Loss: 0.024311592802405357
Epoch: 4431 Loss: 0.02431276999413967
Epoch: 4432 Loss: 0.024295449256896973
Epoch: 4433 Loss: 0.024287350475788116
Epoch: 4434 Loss: 0.024284880608320236
Epoch: 4435 Loss: 0.024260355159640312
Epoch: 4436 Loss: 0.024257143959403038
Epoch: 4437 Loss: 0.024251071736216545
Epoch: 4438 Loss: 0.024236317723989487
Epoch: 4439 Loss: 0.024231547489762306
Epoch: 4440 Loss: 0.02422606758773327
Epoch: 4441 Loss: 0.024208780378103256
Epoch: 4442 Loss: 0.02420225739479065
Epoch: 4443 Loss: 0.02419249340891838
Epoch: 4444 Loss: 0.024179069325327873
Epoch: 4445 Loss: 0.024169303476810455
Epoch: 4446 Loss: 0.024167481809854507
Epoch: 4447 Loss: 0.02414786070585251
Epoch: 4448 Loss: 0.02414623275399208
Epoch: 4449 Loss: 0.024137653410434723
Epoch: 4450 Loss: 0.024123404175043106
Epoch: 4451 Loss: 0.024114785715937614
Epoch: 4452 Loss: 0.02410951629281044
Epoch: 4453 Loss: 0.024101873859763145
Epoch: 4454 Loss: 0.024093812331557274
Epoch: 4455 Loss: 0.024078836664557457
Epoch: 4456 Loss: 0.024070441722869873
Epoch: 4457 Loss: 0.02406313084065914
Epoch: 4458 Loss: 0.024050509557127953
Epoch: 4459 Loss: 0.024046724662184715
Epoch: 4460 Loss: 0.02403291128575802
Epoch: 4461 Loss: 0.02402297779917717
Epoch: 4462 Loss: 0.0240172166377306
Epoch: 4463 Loss: 0.024001402780413628
Epoch: 4464 Loss: 0.023995274677872658
Epoch: 4465 Loss: 0.023984091356396675
Epoch: 4466 Loss: 0.02397780492901802
Epoch: 4467 Loss: 0.023965125903487206
Epoch: 4468 Loss: 0.023961061611771584
Epoch: 4469 Loss: 0.023956626653671265
Epoch: 4470 Loss: 0.02393803559243679
Epoch: 4471 Loss: 0.023931166157126427
Epoch: 4472 Loss: 0.02392035350203514
Epoch: 4473 Loss: 0.023907771334052086
Epoch: 4474 Loss: 0.02390500344336033
Epoch: 4475 Loss: 0.02389499731361866
Epoch: 4476 Loss: 0.023883245885372162
Epoch: 4477 Loss: 0.023875277489423752
Epoch: 4478 Loss: 0.02386818453669548
Epoch: 4479 Loss: 0.023859385401010513
Epoch: 4480 Loss: 0.02385122887790203
Epoch: 4481 Loss: 0.02384314499795437
Epoch: 4482 Loss: 0.023830238729715347
Epoch: 4483 Loss: 0.02381759136915207
Epoch: 4484 Loss: 0.023810239508748055
Epoch: 4485 Loss: 0.023803269490599632
Epoch: 4486 Loss: 0.02378833293914795
Epoch: 4487 Loss: 0.023785844445228577
Epoch: 4488 Loss: 0.023772062733769417
Epoch: 4489 Loss: 0.0237668938934803
Epoch: 4490 Loss: 0.023756738752126694
Epoch: 4491 Loss: 0.023758919909596443
Epoch: 4492 Loss: 0.0237385556101799
Epoch: 4493 Loss: 0.02373342588543892
Epoch: 4494 Loss: 0.02371908910572529
Epoch: 4495 Loss: 0.02371063642203808
Epoch: 4496 Loss: 0.023703765124082565
Epoch: 4497 Loss: 0.023689668625593185
Epoch: 4498 Loss: 0.02367858774960041
Epoch: 4499 Loss: 0.023676984012126923
Epoch: 4500 Loss: 0.023665331304073334
Epoch: 4501 Loss: 0.023653170093894005
Epoch: 4502 Loss: 0.023646192625164986
Epoch: 4503 Loss: 0.023638295009732246
Epoch: 4504 Loss: 0.023633543401956558
Epoch: 4505 Loss: 0.023624924942851067
Epoch: 4506 Loss: 0.023614272475242615
Epoch: 4507 Loss: 0.02360302023589611
Epoch: 4508 Loss: 0.023597056046128273
Epoch: 4509 Loss: 0.023583214730024338
Epoch: 4510 Loss: 0.023579612374305725
Epoch: 4511 Loss: 0.023565495386719704
Epoch: 4512 Loss: 0.02355281263589859
Epoch: 4513 Loss: 0.023545987904071808
Epoch: 4514 Loss: 0.023533731698989868
Epoch: 4515 Loss: 0.023530462756752968
Epoch: 4516 Loss: 0.023529047146439552
Epoch: 4517 Loss: 0.023513225838541985
Epoch: 4518 Loss: 0.023504018783569336
Epoch: 4519 Loss: 0.02349402941763401
Epoch: 4520 Loss: 0.02348512038588524
Epoch: 4521 Loss: 0.023475946858525276
Epoch: 4522 Loss: 0.023466570302844048
Epoch: 4523 Loss: 0.023465771228075027
Epoch: 4524 Loss: 0.02344980090856552
Epoch: 4525 Loss: 0.023439928889274597
Epoch: 4526 Loss: 0.02343287318944931
Epoch: 4527 Loss: 0.023425575345754623
Epoch: 4528 Loss: 0.023412004113197327
Epoch: 4529 Loss: 0.023408658802509308
Epoch: 4530 Loss: 0.02339589037001133
Epoch: 4531 Loss: 0.023389432579278946
Epoch: 4532 Loss: 0.023383062332868576
Epoch: 4533 Loss: 0.023373637348413467
Epoch: 4534 Loss: 0.023360325023531914
Epoch: 4535 Loss: 0.02335432730615139
Epoch: 4536 Loss: 0.023343322798609734
Epoch: 4537 Loss: 0.023332392796874046
Epoch: 4538 Loss: 0.023324940353631973
Epoch: 4539 Loss: 0.023319071158766747
Epoch: 4540 Loss: 0.023309897631406784
Epoch: 4541 Loss: 0.023299239575862885
Epoch: 4542 Loss: 0.02329077385365963
Epoch: 4543 Loss: 0.023281481117010117
Epoch: 4544 Loss: 0.02327384613454342
Epoch: 4545 Loss: 0.023269047960639
Epoch: 4546 Loss: 0.023258794099092484
Epoch: 4547 Loss: 0.02324475534260273
Epoch: 4548 Loss: 0.023237744346261024
Epoch: 4549 Loss: 0.02322842925786972
Epoch: 4550 Loss: 0.023217778652906418
Epoch: 4551 Loss: 0.023213662207126617
Epoch: 4552 Loss: 0.023203540593385696
Epoch: 4553 Loss: 0.023193560540676117
Epoch: 4554 Loss: 0.02318771369755268
Epoch: 4555 Loss: 0.02317703701555729
Epoch: 4556 Loss: 0.02317032404243946
Epoch: 4557 Loss: 0.023157481104135513
Epoch: 4558 Loss: 0.023148827254772186
Epoch: 4559 Loss: 0.023143809288740158
Epoch: 4560 Loss: 0.0231342613697052
Epoch: 4561 Loss: 0.023129969835281372
Epoch: 4562 Loss: 0.02311553992331028
Epoch: 4563 Loss: 0.02310452237725258
Epoch: 4564 Loss: 0.02309654839336872
Epoch: 4565 Loss: 0.023093195632100105
Epoch: 4566 Loss: 0.02307976968586445
Epoch: 4567 Loss: 0.023073580116033554
Epoch: 4568 Loss: 0.023061474785208702
Epoch: 4569 Loss: 0.023060407489538193
Epoch: 4570 Loss: 0.02304651588201523
Epoch: 4571 Loss: 0.023037998005747795
Epoch: 4572 Loss: 0.02302628755569458
Epoch: 4573 Loss: 0.02302164025604725
Epoch: 4574 Loss: 0.023014329373836517
Epoch: 4575 Loss: 0.023002227768301964
Epoch: 4576 Loss: 0.022996224462985992
Epoch: 4577 Loss: 0.022984536364674568
Epoch: 4578 Loss: 0.022977221757173538
Epoch: 4579 Loss: 0.02297055721282959
Epoch: 4580 Loss: 0.022962063550949097
Epoch: 4581 Loss: 0.022948941215872765
Epoch: 4582 Loss: 0.022939713671803474
Epoch: 4583 Loss: 0.02293435111641884
Epoch: 4584 Loss: 0.022923875600099564
Epoch: 4585 Loss: 0.022914478555321693
Epoch: 4586 Loss: 0.022907622158527374
Epoch: 4587 Loss: 0.022897891700267792
Epoch: 4588 Loss: 0.022892938926815987
Epoch: 4589 Loss: 0.02288176119327545
Epoch: 4590 Loss: 0.022872889414429665
Epoch: 4591 Loss: 0.0228619072586298
Epoch: 4592 Loss: 0.022863339632749557
Epoch: 4593 Loss: 0.022845078259706497
Epoch: 4594 Loss: 0.022834351286292076
Epoch: 4595 Loss: 0.022830402478575706
Epoch: 4596 Loss: 0.022826412692666054
Epoch: 4597 Loss: 0.022813256829977036
Epoch: 4598 Loss: 0.022801173850893974
Epoch: 4599 Loss: 0.022794285789132118
Epoch: 4600 Loss: 0.022783316671848297
Epoch: 4601 Loss: 0.022778639569878578
Epoch: 4602 Loss: 0.022767553105950356
Epoch: 4603 Loss: 0.022761918604373932
Epoch: 4604 Loss: 0.022753966972231865
Epoch: 4605 Loss: 0.02275424636900425
Epoch: 4606 Loss: 0.0227345060557127
Epoch: 4607 Loss: 0.022728532552719116
Epoch: 4608 Loss: 0.022718384861946106
Epoch: 4609 Loss: 0.0227121002972126
Epoch: 4610 Loss: 0.022704944014549255
Epoch: 4611 Loss: 0.022692084312438965
Epoch: 4612 Loss: 0.02268706075847149
Epoch: 4613 Loss: 0.02267327345907688
Epoch: 4614 Loss: 0.02267344295978546
Epoch: 4615 Loss: 0.02266266755759716
Epoch: 4616 Loss: 0.022653253749012947
Epoch: 4617 Loss: 0.022648530080914497
Epoch: 4618 Loss: 0.022635413333773613
Epoch: 4619 Loss: 0.022624274715781212
Epoch: 4620 Loss: 0.02261829562485218
Epoch: 4621 Loss: 0.022610528394579887
Epoch: 4622 Loss: 0.022600270807743073
Epoch: 4623 Loss: 0.022599929943680763
Epoch: 4624 Loss: 0.022589314728975296
Epoch: 4625 Loss: 0.022574637085199356
Epoch: 4626 Loss: 0.022573906928300858
Epoch: 4627 Loss: 0.022559847682714462
Epoch: 4628 Loss: 0.022557303309440613
Epoch: 4629 Loss: 0.02254089154303074
Epoch: 4630 Loss: 0.022535286843776703
Epoch: 4631 Loss: 0.02253156714141369
Epoch: 4632 Loss: 0.022525455802679062
Epoch: 4633 Loss: 0.022509200498461723
Epoch: 4634 Loss: 0.02250049263238907
Epoch: 4635 Loss: 0.022492235526442528
Epoch: 4636 Loss: 0.022487133741378784
Epoch: 4637 Loss: 0.022480254992842674
Epoch: 4638 Loss: 0.022468527778983116
Epoch: 4639 Loss: 0.02245878428220749
Epoch: 4640 Loss: 0.02245197258889675
Epoch: 4641 Loss: 0.022446060553193092
Epoch: 4642 Loss: 0.022438710555434227
Epoch: 4643 Loss: 0.022429514676332474
Epoch: 4644 Loss: 0.022421924397349358
Epoch: 4645 Loss: 0.022411266341805458
Epoch: 4646 Loss: 0.022399449720978737
Epoch: 4647 Loss: 0.022395148873329163
Epoch: 4648 Loss: 0.02238546684384346
Epoch: 4649 Loss: 0.022377416491508484
Epoch: 4650 Loss: 0.02236776053905487
Epoch: 4651 Loss: 0.022365687415003777
Epoch: 4652 Loss: 0.02235519513487816
Epoch: 4653 Loss: 0.022346969693899155
Epoch: 4654 Loss: 0.02233564853668213
Epoch: 4655 Loss: 0.02233239635825157
Epoch: 4656 Loss: 0.022322099655866623
Epoch: 4657 Loss: 0.022315694019198418
Epoch: 4658 Loss: 0.022301431745290756
Epoch: 4659 Loss: 0.0222935788333416
Epoch: 4660 Loss: 0.02228979952633381
Epoch: 4661 Loss: 0.022276978939771652
Epoch: 4662 Loss: 0.02227148972451687
Epoch: 4663 Loss: 0.02226494811475277
Epoch: 4664 Loss: 0.022256698459386826
Epoch: 4665 Loss: 0.022256040945649147
Epoch: 4666 Loss: 0.022239569574594498
Epoch: 4667 Loss: 0.022226303815841675
Epoch: 4668 Loss: 0.022220196202397346
Epoch: 4669 Loss: 0.022209137678146362
Epoch: 4670 Loss: 0.022206610068678856
Epoch: 4671 Loss: 0.02219627983868122
Epoch: 4672 Loss: 0.022186560556292534
Epoch: 4673 Loss: 0.02218656614422798
Epoch: 4674 Loss: 0.022169696167111397
Epoch: 4675 Loss: 0.02216479927301407
Epoch: 4676 Loss: 0.022159626707434654
Epoch: 4677 Loss: 0.02214602380990982
Epoch: 4678 Loss: 0.02213914692401886
Epoch: 4679 Loss: 0.022135138511657715
Epoch: 4680 Loss: 0.022121399641036987
Epoch: 4681 Loss: 0.02211279608309269
Epoch: 4682 Loss: 0.022108620032668114
Epoch: 4683 Loss: 0.02209651656448841
Epoch: 4684 Loss: 0.022090550512075424
Epoch: 4685 Loss: 0.02208566851913929
Epoch: 4686 Loss: 0.02207515761256218
Epoch: 4687 Loss: 0.0220720823854208
Epoch: 4688 Loss: 0.02205437608063221
Epoch: 4689 Loss: 0.022051643580198288
Epoch: 4690 Loss: 0.022044671699404716
Epoch: 4691 Loss: 0.022034846246242523
Epoch: 4692 Loss: 0.022027011960744858
Epoch: 4693 Loss: 0.0220146793872118
Epoch: 4694 Loss: 0.02201138064265251
Epoch: 4695 Loss: 0.022001972422003746
Epoch: 4696 Loss: 0.021994788199663162
Epoch: 4697 Loss: 0.021988704800605774
Epoch: 4698 Loss: 0.021981624886393547
Epoch: 4699 Loss: 0.021970001980662346
Epoch: 4700 Loss: 0.021965382620692253
Epoch: 4701 Loss: 0.021950216963887215
Epoch: 4702 Loss: 0.021950738504529
Epoch: 4703 Loss: 0.021935168653726578
Epoch: 4704 Loss: 0.021928388625383377
Epoch: 4705 Loss: 0.021918298676609993
Epoch: 4706 Loss: 0.021911075338721275
Epoch: 4707 Loss: 0.021901782602071762
Epoch: 4708 Loss: 0.021892966702580452
Epoch: 4709 Loss: 0.02188921719789505
Epoch: 4710 Loss: 0.021883051842451096
Epoch: 4711 Loss: 0.021869396790862083
Epoch: 4712 Loss: 0.021860193461179733
Epoch: 4713 Loss: 0.021851947531104088
Epoch: 4714 Loss: 0.021851081401109695
Epoch: 4715 Loss: 0.021838512271642685
Epoch: 4716 Loss: 0.021840838715434074
Epoch: 4717 Loss: 0.021825192496180534
Epoch: 4718 Loss: 0.021820416674017906
Epoch: 4719 Loss: 0.021807007491588593
Epoch: 4720 Loss: 0.021804803982377052
Epoch: 4721 Loss: 0.021792735904455185
Epoch: 4722 Loss: 0.02178698591887951
Epoch: 4723 Loss: 0.02177693322300911
Epoch: 4724 Loss: 0.02177685685455799
Epoch: 4725 Loss: 0.021761320531368256
Epoch: 4726 Loss: 0.021752314642071724
Epoch: 4727 Loss: 0.021745020523667336
Epoch: 4728 Loss: 0.021739354357123375
Epoch: 4729 Loss: 0.021731499582529068
Epoch: 4730 Loss: 0.02172352746129036
Epoch: 4731 Loss: 0.02171047404408455
Epoch: 4732 Loss: 0.021710406988859177
Epoch: 4733 Loss: 0.021698471158742905
Epoch: 4734 Loss: 0.021691754460334778
Epoch: 4735 Loss: 0.021687671542167664
Epoch: 4736 Loss: 0.02167539857327938
Epoch: 4737 Loss: 0.021667899563908577
Epoch: 4738 Loss: 0.021656692028045654
Epoch: 4739 Loss: 0.02165468968451023
Epoch: 4740 Loss: 0.021641623228788376
Epoch: 4741 Loss: 0.021633414551615715
Epoch: 4742 Loss: 0.02163231559097767
Epoch: 4743 Loss: 0.02161846123635769
Epoch: 4744 Loss: 0.0216108076274395
Epoch: 4745 Loss: 0.021608661860227585
Epoch: 4746 Loss: 0.021597368642687798
Epoch: 4747 Loss: 0.02159210853278637
Epoch: 4748 Loss: 0.02157985419034958
Epoch: 4749 Loss: 0.021572917699813843
Epoch: 4750 Loss: 0.021564066410064697
Epoch: 4751 Loss: 0.021559692919254303
Epoch: 4752 Loss: 0.02155163139104843
Epoch: 4753 Loss: 0.021543070673942566
Epoch: 4754 Loss: 0.021537357941269875
Epoch: 4755 Loss: 0.021530579775571823
Epoch: 4756 Loss: 0.021518884226679802
Epoch: 4757 Loss: 0.021507544443011284
Epoch: 4758 Loss: 0.021503979340195656
Epoch: 4759 Loss: 0.021496493369340897
Epoch: 4760 Loss: 0.021485760807991028
Epoch: 4761 Loss: 0.021483397111296654
Epoch: 4762 Loss: 0.02146906591951847
Epoch: 4763 Loss: 0.021467335522174835
Epoch: 4764 Loss: 0.021460644900798798
Epoch: 4765 Loss: 0.021449973806738853
Epoch: 4766 Loss: 0.02143896371126175
Epoch: 4767 Loss: 0.021434366703033447
Epoch: 4768 Loss: 0.021427251398563385
Epoch: 4769 Loss: 0.021418243646621704
Epoch: 4770 Loss: 0.021413249894976616
Epoch: 4771 Loss: 0.021400945261120796
Epoch: 4772 Loss: 0.021398581564426422
Epoch: 4773 Loss: 0.021393947303295135
Epoch: 4774 Loss: 0.021381201222538948
Epoch: 4775 Loss: 0.021371640264987946
Epoch: 4776 Loss: 0.021365361288189888
Epoch: 4777 Loss: 0.021358653903007507
Epoch: 4778 Loss: 0.021346095949411392
Epoch: 4779 Loss: 0.021342450752854347
Epoch: 4780 Loss: 0.021336432546377182
Epoch: 4781 Loss: 0.021325109526515007
Epoch: 4782 Loss: 0.021316152065992355
Epoch: 4783 Loss: 0.021312130615115166
Epoch: 4784 Loss: 0.021310027688741684
Epoch: 4785 Loss: 0.021291866898536682
Epoch: 4786 Loss: 0.021288394927978516
Epoch: 4787 Loss: 0.021287433803081512
Epoch: 4788 Loss: 0.021272655576467514
Epoch: 4789 Loss: 0.02126612514257431
Epoch: 4790 Loss: 0.02125747874379158
Epoch: 4791 Loss: 0.02124822326004505
Epoch: 4792 Loss: 0.02124890871345997
Epoch: 4793 Loss: 0.021234476938843727
Epoch: 4794 Loss: 0.02122724987566471
Epoch: 4795 Loss: 0.021222317591309547
Epoch: 4796 Loss: 0.021215008571743965
Epoch: 4797 Loss: 0.021201618015766144
Epoch: 4798 Loss: 0.02120065502822399
Epoch: 4799 Loss: 0.021193061023950577
Epoch: 4800 Loss: 0.021183878183364868
Epoch: 4801 Loss: 0.021175367757678032
Epoch: 4802 Loss: 0.021166766062378883
Epoch: 4803 Loss: 0.02115941233932972
Epoch: 4804 Loss: 0.021151075139641762
Epoch: 4805 Loss: 0.021141808480024338
Epoch: 4806 Loss: 0.021135563030838966
Epoch: 4807 Loss: 0.021133707836270332
Epoch: 4808 Loss: 0.021120497956871986
Epoch: 4809 Loss: 0.02111244946718216
Epoch: 4810 Loss: 0.021105431020259857
Epoch: 4811 Loss: 0.021101418882608414
Epoch: 4812 Loss: 0.021093901246786118
Epoch: 4813 Loss: 0.021085798740386963
Epoch: 4814 Loss: 0.021075934171676636
Epoch: 4815 Loss: 0.021072987467050552
Epoch: 4816 Loss: 0.02106293849647045
Epoch: 4817 Loss: 0.021054567769169807
Epoch: 4818 Loss: 0.02105029672384262
Epoch: 4819 Loss: 0.021040456369519234
Epoch: 4820 Loss: 0.021036019548773766
Epoch: 4821 Loss: 0.021028798073530197
Epoch: 4822 Loss: 0.021020201966166496
Epoch: 4823 Loss: 0.02100774645805359
Epoch: 4824 Loss: 0.020999476313591003
Epoch: 4825 Loss: 0.020992405712604523
Epoch: 4826 Loss: 0.020985102280974388
Epoch: 4827 Loss: 0.020979762077331543
Epoch: 4828 Loss: 0.020970789715647697
Epoch: 4829 Loss: 0.02096373215317726
Epoch: 4830 Loss: 0.020956970751285553
Epoch: 4831 Loss: 0.020949576050043106
Epoch: 4832 Loss: 0.02094772830605507
Epoch: 4833 Loss: 0.02094174176454544
Epoch: 4834 Loss: 0.02093077078461647
Epoch: 4835 Loss: 0.020924953743815422
Epoch: 4836 Loss: 0.020915905013680458
Epoch: 4837 Loss: 0.02090603858232498
Epoch: 4838 Loss: 0.020897826179862022
Epoch: 4839 Loss: 0.020893702283501625
Epoch: 4840 Loss: 0.020885808393359184
Epoch: 4841 Loss: 0.02087477594614029
Epoch: 4842 Loss: 0.020877249538898468
Epoch: 4843 Loss: 0.020866645500063896
Epoch: 4844 Loss: 0.020855195820331573
Epoch: 4845 Loss: 0.020849473774433136
Epoch: 4846 Loss: 0.02083839289844036
Epoch: 4847 Loss: 0.020835263654589653
Epoch: 4848 Loss: 0.020825790241360664
Epoch: 4849 Loss: 0.020826848223805428
Epoch: 4850 Loss: 0.020810643211007118
Epoch: 4851 Loss: 0.020807072520256042
Epoch: 4852 Loss: 0.020802035927772522
Epoch: 4853 Loss: 0.02079404890537262
Epoch: 4854 Loss: 0.020782791078090668
Epoch: 4855 Loss: 0.020783722400665283
Epoch: 4856 Loss: 0.020766034722328186
Epoch: 4857 Loss: 0.02076282538473606
Epoch: 4858 Loss: 0.020753193646669388
Epoch: 4859 Loss: 0.020745737478137016
Epoch: 4860 Loss: 0.020742300897836685
Epoch: 4861 Loss: 0.02073080837726593
Epoch: 4862 Loss: 0.020728981122374535
Epoch: 4863 Loss: 0.02072005718946457
Epoch: 4864 Loss: 0.020713673904538155
Epoch: 4865 Loss: 0.020705802366137505
Epoch: 4866 Loss: 0.02069702185690403
Epoch: 4867 Loss: 0.020692933350801468
Epoch: 4868 Loss: 0.02067979983985424
Epoch: 4869 Loss: 0.02067825198173523
Epoch: 4870 Loss: 0.020670350641012192
Epoch: 4871 Loss: 0.020659737288951874
Epoch: 4872 Loss: 0.02065873332321644
Epoch: 4873 Loss: 0.02064482308924198
Epoch: 4874 Loss: 0.02063809148967266
Epoch: 4875 Loss: 0.02063361369073391
Epoch: 4876 Loss: 0.02062671259045601
Epoch: 4877 Loss: 0.020618947222828865
Epoch: 4878 Loss: 0.020611001178622246
Epoch: 4879 Loss: 0.020607637241482735
Epoch: 4880 Loss: 0.02059909515082836
Epoch: 4881 Loss: 0.020590901374816895
Epoch: 4882 Loss: 0.020585637539625168
Epoch: 4883 Loss: 0.02057141438126564
Epoch: 4884 Loss: 0.0205709058791399
Epoch: 4885 Loss: 0.02055669017136097
Epoch: 4886 Loss: 0.020554320886731148
Epoch: 4887 Loss: 0.0205458402633667
Epoch: 4888 Loss: 0.02053968235850334
Epoch: 4889 Loss: 0.020530883222818375
Epoch: 4890 Loss: 0.020529882982373238
Epoch: 4891 Loss: 0.02051933854818344
Epoch: 4892 Loss: 0.020514629781246185
Epoch: 4893 Loss: 0.020505830645561218
Epoch: 4894 Loss: 0.020498011261224747
Epoch: 4895 Loss: 0.020493270829319954
Epoch: 4896 Loss: 0.02048727683722973
Epoch: 4897 Loss: 0.020475631579756737
Epoch: 4898 Loss: 0.020468397065997124
Epoch: 4899 Loss: 0.02046319656074047
Epoch: 4900 Loss: 0.020458992570638657
Epoch: 4901 Loss: 0.020452752709388733
Epoch: 4902 Loss: 0.020441675558686256
Epoch: 4903 Loss: 0.02043449506163597
Epoch: 4904 Loss: 0.020423607900738716
Epoch: 4905 Loss: 0.02042190358042717
Epoch: 4906 Loss: 0.02041434869170189
Epoch: 4907 Loss: 0.0204024575650692
Epoch: 4908 Loss: 0.020400574430823326
Epoch: 4909 Loss: 0.020393479615449905
Epoch: 4910 Loss: 0.02038855478167534
Epoch: 4911 Loss: 0.020379837602376938
Epoch: 4912 Loss: 0.020369984209537506
Epoch: 4913 Loss: 0.020366361364722252
Epoch: 4914 Loss: 0.020359642803668976
Epoch: 4915 Loss: 0.02035408653318882
Epoch: 4916 Loss: 0.020340343937277794
Epoch: 4917 Loss: 0.02033538743853569
Epoch: 4918 Loss: 0.020325876772403717
Epoch: 4919 Loss: 0.02032512053847313
Epoch: 4920 Loss: 0.02031751535832882
Epoch: 4921 Loss: 0.020305210724473
Epoch: 4922 Loss: 0.02030680887401104
Epoch: 4923 Loss: 0.02029530331492424
Epoch: 4924 Loss: 0.02028576470911503
Epoch: 4925 Loss: 0.020281154662370682
Epoch: 4926 Loss: 0.0202745720744133
Epoch: 4927 Loss: 0.020267195999622345
Epoch: 4928 Loss: 0.020259426906704903
Epoch: 4929 Loss: 0.02025808021426201
Epoch: 4930 Loss: 0.020244093611836433
Epoch: 4931 Loss: 0.020239371806383133
Epoch: 4932 Loss: 0.02023095265030861
Epoch: 4933 Loss: 0.020230887457728386
Epoch: 4934 Loss: 0.020216574892401695
Epoch: 4935 Loss: 0.02020956017076969
Epoch: 4936 Loss: 0.020204035565257072
Epoch: 4937 Loss: 0.020199976861476898
Epoch: 4938 Loss: 0.020190058276057243
Epoch: 4939 Loss: 0.020182903856039047
Epoch: 4940 Loss: 0.020176514983177185
Epoch: 4941 Loss: 0.02016734890639782
Epoch: 4942 Loss: 0.02016696333885193
Epoch: 4943 Loss: 0.02015691250562668
Epoch: 4944 Loss: 0.020148318260908127
Epoch: 4945 Loss: 0.02014734409749508
Epoch: 4946 Loss: 0.02013852447271347
Epoch: 4947 Loss: 0.020128680393099785
Epoch: 4948 Loss: 0.02012001723051071
Epoch: 4949 Loss: 0.020116465166211128
Epoch: 4950 Loss: 0.020108662545681
Epoch: 4951 Loss: 0.020103517919778824
Epoch: 4952 Loss: 0.02009611390531063
Epoch: 4953 Loss: 0.020090317353606224
Epoch: 4954 Loss: 0.02008374221622944
Epoch: 4955 Loss: 0.020072750747203827
Epoch: 4956 Loss: 0.02007090486586094
Epoch: 4957 Loss: 0.020062172785401344
Epoch: 4958 Loss: 0.020053327083587646
Epoch: 4959 Loss: 0.020048029720783234
Epoch: 4960 Loss: 0.02004115656018257
Epoch: 4961 Loss: 0.02003978006541729
Epoch: 4962 Loss: 0.020026782527565956
Epoch: 4963 Loss: 0.020018745213747025
Epoch: 4964 Loss: 0.020015012472867966
Epoch: 4965 Loss: 0.020009197294712067
Epoch: 4966 Loss: 0.020002711564302444
Epoch: 4967 Loss: 0.01999579928815365
Epoch: 4968 Loss: 0.019988220185041428
Epoch: 4969 Loss: 0.01998126320540905
Epoch: 4970 Loss: 0.01997152902185917
Epoch: 4971 Loss: 0.019965633749961853
Epoch: 4972 Loss: 0.019959408789873123
Epoch: 4973 Loss: 0.019959421828389168
Epoch: 4974 Loss: 0.019946765154600143
Epoch: 4975 Loss: 0.0199427530169487
Epoch: 4976 Loss: 0.019933827221393585
Epoch: 4977 Loss: 0.019927222281694412
Epoch: 4978 Loss: 0.019923340529203415
Epoch: 4979 Loss: 0.019915197044610977
Epoch: 4980 Loss: 0.019906947389245033
Epoch: 4981 Loss: 0.01989840529859066
Epoch: 4982 Loss: 0.01989581063389778
Epoch: 4983 Loss: 0.019886493682861328
Epoch: 4984 Loss: 0.019888298586010933
Epoch: 4985 Loss: 0.019871313124895096
Epoch: 4986 Loss: 0.01986818201839924
Epoch: 4987 Loss: 0.019857387989759445
Epoch: 4988 Loss: 0.01985648274421692
Epoch: 4989 Loss: 0.01984359510242939
Epoch: 4990 Loss: 0.01983960159122944
Epoch: 4991 Loss: 0.019838400185108185
Epoch: 4992 Loss: 0.019827984273433685
Epoch: 4993 Loss: 0.019819272682070732
Epoch: 4994 Loss: 0.019808927550911903
Epoch: 4995 Loss: 0.019802821800112724
Epoch: 4996 Loss: 0.01979723758995533
Epoch: 4997 Loss: 0.019794689491391182
Epoch: 4998 Loss: 0.0197825338691473
Epoch: 4999 Loss: 0.01977512799203396
Epoch: 5000 Loss: 0.019772158935666084
Epoch: 5001 Loss: 0.019761933013796806
Epoch: 5002 Loss: 0.019754590466618538
Epoch: 5003 Loss: 0.01974637247622013
Epoch: 5004 Loss: 0.019749771803617477
Epoch: 5005 Loss: 0.019736427813768387
Epoch: 5006 Loss: 0.019728653132915497
Epoch: 5007 Loss: 0.019722681492567062
Epoch: 5008 Loss: 0.019714320078492165
Epoch: 5009 Loss: 0.019709570333361626
Epoch: 5010 Loss: 0.01970032788813114
Epoch: 5011 Loss: 0.019691694527864456
Epoch: 5012 Loss: 0.01968975178897381
Epoch: 5013 Loss: 0.019683057442307472
Epoch: 5014 Loss: 0.019673602655529976
Epoch: 5015 Loss: 0.01966780610382557
Epoch: 5016 Loss: 0.019659029319882393
Epoch: 5017 Loss: 0.019653398543596268
Epoch: 5018 Loss: 0.019646773114800453
Epoch: 5019 Loss: 0.01964321918785572
Epoch: 5020 Loss: 0.019630733877420425
Epoch: 5021 Loss: 0.019625181332230568
Epoch: 5022 Loss: 0.01962256245315075
Epoch: 5023 Loss: 0.01961088366806507
Epoch: 5024 Loss: 0.019602572545409203
Epoch: 5025 Loss: 0.019598698243498802
Epoch: 5026 Loss: 0.019590074196457863
Epoch: 5027 Loss: 0.01958337612450123
Epoch: 5028 Loss: 0.01957874372601509
Epoch: 5029 Loss: 0.019572053104639053
Epoch: 5030 Loss: 0.01956641487777233
Epoch: 5031 Loss: 0.019557613879442215
Epoch: 5032 Loss: 0.019549770280718803
Epoch: 5033 Loss: 0.019547371193766594
Epoch: 5034 Loss: 0.01953703910112381
Epoch: 5035 Loss: 0.019529830664396286
Epoch: 5036 Loss: 0.01952599361538887
Epoch: 5037 Loss: 0.01951976679265499
Epoch: 5038 Loss: 0.019513875246047974
Epoch: 5039 Loss: 0.019505584612488747
Epoch: 5040 Loss: 0.019497811794281006
Epoch: 5041 Loss: 0.01949029043316841
Epoch: 5042 Loss: 0.019483324140310287
Epoch: 5043 Loss: 0.019475547596812248
Epoch: 5044 Loss: 0.019470348954200745
Epoch: 5045 Loss: 0.01946946233510971
Epoch: 5046 Loss: 0.019456373527646065
Epoch: 5047 Loss: 0.019452787935733795
Epoch: 5048 Loss: 0.019443713128566742
Epoch: 5049 Loss: 0.019438734278082848
Epoch: 5050 Loss: 0.019432391971349716
Epoch: 5051 Loss: 0.01942335069179535
Epoch: 5052 Loss: 0.01941942609846592
Epoch: 5053 Loss: 0.019415564835071564
Epoch: 5054 Loss: 0.019401172176003456
Epoch: 5055 Loss: 0.01939743384718895
Epoch: 5056 Loss: 0.019389647990465164
Epoch: 5057 Loss: 0.019382407888770103
Epoch: 5058 Loss: 0.019376656040549278
Epoch: 5059 Loss: 0.019369546324014664
Epoch: 5060 Loss: 0.019369902089238167
Epoch: 5061 Loss: 0.01935642398893833
Epoch: 5062 Loss: 0.019352801144123077
Epoch: 5063 Loss: 0.019344210624694824
Epoch: 5064 Loss: 0.01933852769434452
Epoch: 5065 Loss: 0.019328096881508827
Epoch: 5066 Loss: 0.01932746171951294
Epoch: 5067 Loss: 0.019318781793117523
Epoch: 5068 Loss: 0.019315529614686966
Epoch: 5069 Loss: 0.019304439425468445
Epoch: 5070 Loss: 0.01929791085422039
Epoch: 5071 Loss: 0.019293980672955513
Epoch: 5072 Loss: 0.019288426265120506
Epoch: 5073 Loss: 0.01927664875984192
Epoch: 5074 Loss: 0.019269656389951706
Epoch: 5075 Loss: 0.01926434598863125
Epoch: 5076 Loss: 0.01925782486796379
Epoch: 5077 Loss: 0.019252479076385498
Epoch: 5078 Loss: 0.019251452758908272
Epoch: 5079 Loss: 0.019241269677877426
Epoch: 5080 Loss: 0.01923041231930256
Epoch: 5081 Loss: 0.019224023446440697
Epoch: 5082 Loss: 0.019222116097807884
Epoch: 5083 Loss: 0.019213451072573662
Epoch: 5084 Loss: 0.019211461767554283
Epoch: 5085 Loss: 0.01919681392610073
Epoch: 5086 Loss: 0.019193826243281364
Epoch: 5087 Loss: 0.01918746344745159
Epoch: 5088 Loss: 0.019180281087756157
Epoch: 5089 Loss: 0.019175078719854355
Epoch: 5090 Loss: 0.019173800945281982
Epoch: 5091 Loss: 0.01916017383337021
Epoch: 5092 Loss: 0.019153956323862076
Epoch: 5093 Loss: 0.019147424027323723
Epoch: 5094 Loss: 0.019142011180520058
Epoch: 5095 Loss: 0.019133230671286583
Epoch: 5096 Loss: 0.019127778708934784
Epoch: 5097 Loss: 0.019124282523989677
Epoch: 5098 Loss: 0.01911802962422371
Epoch: 5099 Loss: 0.019109340384602547
Epoch: 5100 Loss: 0.01910664141178131
Epoch: 5101 Loss: 0.019098147749900818
Epoch: 5102 Loss: 0.0190891083329916
Epoch: 5103 Loss: 0.019080737605690956
Epoch: 5104 Loss: 0.019076494500041008
Epoch: 5105 Loss: 0.01907380111515522
Epoch: 5106 Loss: 0.019062310457229614
Epoch: 5107 Loss: 0.019057035446166992
Epoch: 5108 Loss: 0.019050534814596176
Epoch: 5109 Loss: 0.019041633233428
Epoch: 5110 Loss: 0.01903611235320568
Epoch: 5111 Loss: 0.019031794741749763
Epoch: 5112 Loss: 0.019028617069125175
Epoch: 5113 Loss: 0.019019316881895065
Epoch: 5114 Loss: 0.019016435369849205
Epoch: 5115 Loss: 0.01900506019592285
Epoch: 5116 Loss: 0.019004913046956062
Epoch: 5117 Loss: 0.018992459401488304
Epoch: 5118 Loss: 0.018989427015185356
Epoch: 5119 Loss: 0.018980354070663452
Epoch: 5120 Loss: 0.018976114690303802
Epoch: 5121 Loss: 0.01896798424422741
Epoch: 5122 Loss: 0.018960030749440193
Epoch: 5123 Loss: 0.018954377621412277
Epoch: 5124 Loss: 0.01895102486014366
Epoch: 5125 Loss: 0.0189428199082613
Epoch: 5126 Loss: 0.018937762826681137
Epoch: 5127 Loss: 0.018930139020085335
Epoch: 5128 Loss: 0.018928544595837593
Epoch: 5129 Loss: 0.018916625529527664
Epoch: 5130 Loss: 0.018910696730017662
Epoch: 5131 Loss: 0.018905792385339737
Epoch: 5132 Loss: 0.01889507658779621
Epoch: 5133 Loss: 0.018893534317612648
Epoch: 5134 Loss: 0.018892573192715645
Epoch: 5135 Loss: 0.01888119988143444
Epoch: 5136 Loss: 0.01887478306889534
Epoch: 5137 Loss: 0.018868081271648407
Epoch: 5138 Loss: 0.01885962300002575
Epoch: 5139 Loss: 0.01885201409459114
Epoch: 5140 Loss: 0.01884816214442253
Epoch: 5141 Loss: 0.018843213096261024
Epoch: 5142 Loss: 0.018834687769412994
Epoch: 5143 Loss: 0.018833091482520103
Epoch: 5144 Loss: 0.018823863938450813
Epoch: 5145 Loss: 0.018818313255906105
Epoch: 5146 Loss: 0.01881413161754608
Epoch: 5147 Loss: 0.018802491948008537
Epoch: 5148 Loss: 0.01879732497036457
Epoch: 5149 Loss: 0.01879201829433441
Epoch: 5150 Loss: 0.018782630562782288
Epoch: 5151 Loss: 0.01877760700881481
Epoch: 5152 Loss: 0.018773788586258888
Epoch: 5153 Loss: 0.018769454210996628
Epoch: 5154 Loss: 0.018763964995741844
Epoch: 5155 Loss: 0.01875208131968975
Epoch: 5156 Loss: 0.01874995231628418
Epoch: 5157 Loss: 0.018740255385637283
Epoch: 5158 Loss: 0.01873898319900036
Epoch: 5159 Loss: 0.018726522102952003
Epoch: 5160 Loss: 0.01872207038104534
Epoch: 5161 Loss: 0.018720535561442375
Epoch: 5162 Loss: 0.01870979554951191
Epoch: 5163 Loss: 0.018706288188695908
Epoch: 5164 Loss: 0.0187008585780859
Epoch: 5165 Loss: 0.018690140917897224
Epoch: 5166 Loss: 0.01868877373635769
Epoch: 5167 Loss: 0.01867900975048542
Epoch: 5168 Loss: 0.018674511462450027
Epoch: 5169 Loss: 0.018670622259378433
Epoch: 5170 Loss: 0.01865936629474163
Epoch: 5171 Loss: 0.01865343004465103
Epoch: 5172 Loss: 0.018651220947504044
Epoch: 5173 Loss: 0.018642185255885124
Epoch: 5174 Loss: 0.018634917214512825
Epoch: 5175 Loss: 0.01863202266395092
Epoch: 5176 Loss: 0.01862526312470436
Epoch: 5177 Loss: 0.018616920337080956
Epoch: 5178 Loss: 0.01861206442117691
Epoch: 5179 Loss: 0.018607689067721367
Epoch: 5180 Loss: 0.018597953021526337
Epoch: 5181 Loss: 0.01859721913933754
Epoch: 5182 Loss: 0.018588509410619736
Epoch: 5183 Loss: 0.01858074963092804
Epoch: 5184 Loss: 0.01857750676572323
Epoch: 5185 Loss: 0.018568001687526703
Epoch: 5186 Loss: 0.018562564626336098
Epoch: 5187 Loss: 0.01856342703104019
Epoch: 5188 Loss: 0.018550729379057884
Epoch: 5189 Loss: 0.018544448539614677
Epoch: 5190 Loss: 0.018538525328040123
Epoch: 5191 Loss: 0.01853330433368683
Epoch: 5192 Loss: 0.01852545142173767
Epoch: 5193 Loss: 0.018522920086979866
Epoch: 5194 Loss: 0.018514534458518028
Epoch: 5195 Loss: 0.018512694165110588
Epoch: 5196 Loss: 0.018501659855246544
Epoch: 5197 Loss: 0.018497074022889137
Epoch: 5198 Loss: 0.01849314570426941
Epoch: 5199 Loss: 0.018483145162463188
Epoch: 5200 Loss: 0.01847703568637371
Epoch: 5201 Loss: 0.0184728242456913
Epoch: 5202 Loss: 0.01846315525472164
Epoch: 5203 Loss: 0.018459411337971687
Epoch: 5204 Loss: 0.018454963341355324
Epoch: 5205 Loss: 0.01844925805926323
Epoch: 5206 Loss: 0.01844194531440735
Epoch: 5207 Loss: 0.01843724399805069
Epoch: 5208 Loss: 0.018432732671499252
Epoch: 5209 Loss: 0.018423058092594147
Epoch: 5210 Loss: 0.018421143293380737
Epoch: 5211 Loss: 0.018413664773106575
Epoch: 5212 Loss: 0.018402056768536568
Epoch: 5213 Loss: 0.018397580832242966
Epoch: 5214 Loss: 0.018391793593764305
Epoch: 5215 Loss: 0.01838585175573826
Epoch: 5216 Loss: 0.01838025450706482
Epoch: 5217 Loss: 0.018373792991042137
Epoch: 5218 Loss: 0.018375391140580177
Epoch: 5219 Loss: 0.018363500013947487
Epoch: 5220 Loss: 0.01835770159959793
Epoch: 5221 Loss: 0.018349774181842804
Epoch: 5222 Loss: 0.018348170444369316
Epoch: 5223 Loss: 0.01833919808268547
Epoch: 5224 Loss: 0.01833607815206051
Epoch: 5225 Loss: 0.018330879509449005
Epoch: 5226 Loss: 0.018325556069612503
Epoch: 5227 Loss: 0.018314674496650696
Epoch: 5228 Loss: 0.018309127539396286
Epoch: 5229 Loss: 0.01830250211060047
Epoch: 5230 Loss: 0.01829485036432743
Epoch: 5231 Loss: 0.018292337656021118
Epoch: 5232 Loss: 0.018288537859916687
Epoch: 5233 Loss: 0.01828167401254177
Epoch: 5234 Loss: 0.01827213168144226
Epoch: 5235 Loss: 0.018266230821609497
Epoch: 5236 Loss: 0.01826244406402111
Epoch: 5237 Loss: 0.01825827732682228
Epoch: 5238 Loss: 0.01825566031038761
Epoch: 5239 Loss: 0.01824515126645565
Epoch: 5240 Loss: 0.01824004575610161
Epoch: 5241 Loss: 0.018236085772514343
Epoch: 5242 Loss: 0.018224354833364487
Epoch: 5243 Loss: 0.018222568556666374
Epoch: 5244 Loss: 0.018214279785752296
Epoch: 5245 Loss: 0.018211573362350464
Epoch: 5246 Loss: 0.018199777230620384
Epoch: 5247 Loss: 0.018196867778897285
Epoch: 5248 Loss: 0.018193179741501808
Epoch: 5249 Loss: 0.018188336864113808
Epoch: 5250 Loss: 0.018180783838033676
Epoch: 5251 Loss: 0.018173769116401672
Epoch: 5252 Loss: 0.018169019371271133
Epoch: 5253 Loss: 0.018164971843361855
Epoch: 5254 Loss: 0.018161039799451828
Epoch: 5255 Loss: 0.018151797354221344
Epoch: 5256 Loss: 0.018144547939300537
Epoch: 5257 Loss: 0.018139950931072235
Epoch: 5258 Loss: 0.018136804923415184
Epoch: 5259 Loss: 0.018128683790564537
Epoch: 5260 Loss: 0.018119962885975838
Epoch: 5261 Loss: 0.018119020387530327
Epoch: 5262 Loss: 0.018110334873199463
Epoch: 5263 Loss: 0.018102528527379036
Epoch: 5264 Loss: 0.018099969252943993
Epoch: 5265 Loss: 0.01809650845825672
Epoch: 5266 Loss: 0.018088389188051224
Epoch: 5267 Loss: 0.018081024289131165
Epoch: 5268 Loss: 0.01807897537946701
Epoch: 5269 Loss: 0.018071981146931648
Epoch: 5270 Loss: 0.01806565187871456
Epoch: 5271 Loss: 0.018059715628623962
Epoch: 5272 Loss: 0.01805245690047741
Epoch: 5273 Loss: 0.018048791214823723
Epoch: 5274 Loss: 0.01804075762629509
Epoch: 5275 Loss: 0.018036890774965286
Epoch: 5276 Loss: 0.01803094521164894
Epoch: 5277 Loss: 0.018026921898126602
Epoch: 5278 Loss: 0.018017005175352097
Epoch: 5279 Loss: 0.018010113388299942
Epoch: 5280 Loss: 0.018009649589657784
Epoch: 5281 Loss: 0.017999814823269844
Epoch: 5282 Loss: 0.017996151000261307
Epoch: 5283 Loss: 0.017991099506616592
Epoch: 5284 Loss: 0.017983481287956238
Epoch: 5285 Loss: 0.017977938055992126
Epoch: 5286 Loss: 0.017970874905586243
Epoch: 5287 Loss: 0.017968803644180298
Epoch: 5288 Loss: 0.017964037135243416
Epoch: 5289 Loss: 0.017957676202058792
Epoch: 5290 Loss: 0.017949070781469345
Epoch: 5291 Loss: 0.01794346608221531
Epoch: 5292 Loss: 0.01793736219406128
Epoch: 5293 Loss: 0.017932584509253502
Epoch: 5294 Loss: 0.017926674336194992
Epoch: 5295 Loss: 0.017922380939126015
Epoch: 5296 Loss: 0.0179181806743145
Epoch: 5297 Loss: 0.017909817397594452
Epoch: 5298 Loss: 0.017904579639434814
Epoch: 5299 Loss: 0.01790044829249382
Epoch: 5300 Loss: 0.017894966527819633
Epoch: 5301 Loss: 0.01788962632417679
Epoch: 5302 Loss: 0.017882470041513443
Epoch: 5303 Loss: 0.017879659309983253
Epoch: 5304 Loss: 0.017870329320430756
Epoch: 5305 Loss: 0.017862381413578987
Epoch: 5306 Loss: 0.017860395833849907
Epoch: 5307 Loss: 0.017851650714874268
Epoch: 5308 Loss: 0.017849888652563095
Epoch: 5309 Loss: 0.017842400819063187
Epoch: 5310 Loss: 0.017835328355431557
Epoch: 5311 Loss: 0.01782885752618313
Epoch: 5312 Loss: 0.01782570593059063
Epoch: 5313 Loss: 0.01782054640352726
Epoch: 5314 Loss: 0.017813971266150475
Epoch: 5315 Loss: 0.01780938357114792
Epoch: 5316 Loss: 0.017802756279706955
Epoch: 5317 Loss: 0.017800316214561462
Epoch: 5318 Loss: 0.017788821831345558
Epoch: 5319 Loss: 0.017785590142011642
Epoch: 5320 Loss: 0.017781464383006096
Epoch: 5321 Loss: 0.017775321379303932
Epoch: 5322 Loss: 0.017768077552318573
Epoch: 5323 Loss: 0.017764708027243614
Epoch: 5324 Loss: 0.017758021131157875
Epoch: 5325 Loss: 0.01775239035487175
Epoch: 5326 Loss: 0.017752015963196754
Epoch: 5327 Loss: 0.01773941144347191
Epoch: 5328 Loss: 0.01773745007812977
Epoch: 5329 Loss: 0.0177321657538414
Epoch: 5330 Loss: 0.017723126336932182
Epoch: 5331 Loss: 0.01771596074104309
Epoch: 5332 Loss: 0.017713231965899467
Epoch: 5333 Loss: 0.017706461250782013
Epoch: 5334 Loss: 0.01770070195198059
Epoch: 5335 Loss: 0.017698105424642563
Epoch: 5336 Loss: 0.017691180109977722
Epoch: 5337 Loss: 0.01768869161605835
Epoch: 5338 Loss: 0.0176822692155838
Epoch: 5339 Loss: 0.017674066126346588
Epoch: 5340 Loss: 0.017668869346380234
Epoch: 5341 Loss: 0.017667166888713837
Epoch: 5342 Loss: 0.017653493210673332
Epoch: 5343 Loss: 0.017653513699769974
Epoch: 5344 Loss: 0.01764550246298313
Epoch: 5345 Loss: 0.01764133758842945
Epoch: 5346 Loss: 0.017635634168982506
Epoch: 5347 Loss: 0.017629913985729218
Epoch: 5348 Loss: 0.017626263201236725
Epoch: 5349 Loss: 0.017617126926779747
Epoch: 5350 Loss: 0.017611948773264885
Epoch: 5351 Loss: 0.01761016622185707
Epoch: 5352 Loss: 0.017601629719138145
Epoch: 5353 Loss: 0.017596997320652008
Epoch: 5354 Loss: 0.017595630139112473
Epoch: 5355 Loss: 0.017587196081876755
Epoch: 5356 Loss: 0.01757945865392685
Epoch: 5357 Loss: 0.0175766758620739
Epoch: 5358 Loss: 0.01757068745791912
Epoch: 5359 Loss: 0.017565295100212097
Epoch: 5360 Loss: 0.017559168860316277
Epoch: 5361 Loss: 0.017552291974425316
Epoch: 5362 Loss: 0.017549630254507065
Epoch: 5363 Loss: 0.017543762922286987
Epoch: 5364 Loss: 0.01753714308142662
Epoch: 5365 Loss: 0.017529882490634918
Epoch: 5366 Loss: 0.017525386065244675
Epoch: 5367 Loss: 0.017523283138871193
Epoch: 5368 Loss: 0.01751742884516716
Epoch: 5369 Loss: 0.017512694001197815
Epoch: 5370 Loss: 0.017503349110484123
Epoch: 5371 Loss: 0.01750267669558525
Epoch: 5372 Loss: 0.01749200001358986
Epoch: 5373 Loss: 0.017487188801169395
Epoch: 5374 Loss: 0.01748182624578476
Epoch: 5375 Loss: 0.017479129135608673
Epoch: 5376 Loss: 0.01747088134288788
Epoch: 5377 Loss: 0.01746552437543869
Epoch: 5378 Loss: 0.017461221665143967
Epoch: 5379 Loss: 0.017457114532589912
Epoch: 5380 Loss: 0.017449818551540375
Epoch: 5381 Loss: 0.01744990237057209
Epoch: 5382 Loss: 0.01744314283132553
Epoch: 5383 Loss: 0.01743430271744728
Epoch: 5384 Loss: 0.01742563210427761
Epoch: 5385 Loss: 0.01742035709321499
Epoch: 5386 Loss: 0.017417941242456436
Epoch: 5387 Loss: 0.017415007576346397
Epoch: 5388 Loss: 0.017410054802894592
Epoch: 5389 Loss: 0.01740126498043537
Epoch: 5390 Loss: 0.017395587638020515
Epoch: 5391 Loss: 0.01738656684756279
Epoch: 5392 Loss: 0.017385214567184448
Epoch: 5393 Loss: 0.01738186925649643
Epoch: 5394 Loss: 0.0173761285841465
Epoch: 5395 Loss: 0.017371011897921562
Epoch: 5396 Loss: 0.01736328937113285
Epoch: 5397 Loss: 0.017358051612973213
Epoch: 5398 Loss: 0.017353838309645653
Epoch: 5399 Loss: 0.017345935106277466
Epoch: 5400 Loss: 0.01734176091849804
Epoch: 5401 Loss: 0.017336687073111534
Epoch: 5402 Loss: 0.017335714772343636
Epoch: 5403 Loss: 0.017325803637504578
Epoch: 5404 Loss: 0.0173215139657259
Epoch: 5405 Loss: 0.01731543056666851
Epoch: 5406 Loss: 0.017313499003648758
Epoch: 5407 Loss: 0.017302772030234337
Epoch: 5408 Loss: 0.017299558967351913
Epoch: 5409 Loss: 0.017293371260166168
Epoch: 5410 Loss: 0.017290348187088966
Epoch: 5411 Loss: 0.017281297594308853
Epoch: 5412 Loss: 0.01727611944079399
Epoch: 5413 Loss: 0.01727404072880745
Epoch: 5414 Loss: 0.017267301678657532
Epoch: 5415 Loss: 0.017262756824493408
Epoch: 5416 Loss: 0.017256975173950195
Epoch: 5417 Loss: 0.017250632867217064
Epoch: 5418 Loss: 0.017246689647436142
Epoch: 5419 Loss: 0.017240865156054497
Epoch: 5420 Loss: 0.01723775640130043
Epoch: 5421 Loss: 0.017231477424502373
Epoch: 5422 Loss: 0.017222026363015175
Epoch: 5423 Loss: 0.017217539250850677
Epoch: 5424 Loss: 0.01721375063061714
Epoch: 5425 Loss: 0.017211684957146645
Epoch: 5426 Loss: 0.017201576381921768
Epoch: 5427 Loss: 0.017197472974658012
Epoch: 5428 Loss: 0.017197802662849426
Epoch: 5429 Loss: 0.017186522483825684
Epoch: 5430 Loss: 0.017183316871523857
Epoch: 5431 Loss: 0.017175419256091118
Epoch: 5432 Loss: 0.01717096008360386
Epoch: 5433 Loss: 0.017167799174785614
Epoch: 5434 Loss: 0.0171632282435894
Epoch: 5435 Loss: 0.017160288989543915
Epoch: 5436 Loss: 0.017149241641163826
Epoch: 5437 Loss: 0.017148787155747414
Epoch: 5438 Loss: 0.017139676958322525
Epoch: 5439 Loss: 0.01713493838906288
Epoch: 5440 Loss: 0.017128949984908104
Epoch: 5441 Loss: 0.017123091965913773
Epoch: 5442 Loss: 0.01712164096534252
Epoch: 5443 Loss: 0.017111392691731453
Epoch: 5444 Loss: 0.017109893262386322
Epoch: 5445 Loss: 0.0171041339635849
Epoch: 5446 Loss: 0.017097031697630882
Epoch: 5447 Loss: 0.017092442139983177
Epoch: 5448 Loss: 0.017088716849684715
Epoch: 5449 Loss: 0.01708388701081276
Epoch: 5450 Loss: 0.017079463228583336
Epoch: 5451 Loss: 0.017071709036827087
Epoch: 5452 Loss: 0.0170716755092144
Epoch: 5453 Loss: 0.01706414669752121
Epoch: 5454 Loss: 0.017057688906788826
Epoch: 5455 Loss: 0.01705217733979225
Epoch: 5456 Loss: 0.01704518310725689
Epoch: 5457 Loss: 0.017040353268384933
Epoch: 5458 Loss: 0.017033761367201805
Epoch: 5459 Loss: 0.0170297734439373
Epoch: 5460 Loss: 0.017027420923113823
Epoch: 5461 Loss: 0.01702140085399151
Epoch: 5462 Loss: 0.01701754331588745
Epoch: 5463 Loss: 0.017011823132634163
Epoch: 5464 Loss: 0.017007235437631607
Epoch: 5465 Loss: 0.017002208158373833
Epoch: 5466 Loss: 0.016998032107949257
Epoch: 5467 Loss: 0.01698797009885311
Epoch: 5468 Loss: 0.01698222942650318
Epoch: 5469 Loss: 0.016977591440081596
Epoch: 5470 Loss: 0.016972076147794724
Epoch: 5471 Loss: 0.016967033967375755
Epoch: 5472 Loss: 0.01696278154850006
Epoch: 5473 Loss: 0.016962651163339615
Epoch: 5474 Loss: 0.01695355959236622
Epoch: 5475 Loss: 0.016947586089372635
Epoch: 5476 Loss: 0.01694197580218315
Epoch: 5477 Loss: 0.01694057695567608
Epoch: 5478 Loss: 0.016931746155023575
Epoch: 5479 Loss: 0.016928905621170998
Epoch: 5480 Loss: 0.016922466456890106
Epoch: 5481 Loss: 0.01691424287855625
Epoch: 5482 Loss: 0.016911908984184265
Epoch: 5483 Loss: 0.016905995085835457
Epoch: 5484 Loss: 0.016903281211853027
Epoch: 5485 Loss: 0.01689954660832882
Epoch: 5486 Loss: 0.016892017796635628
Epoch: 5487 Loss: 0.016887294128537178
Epoch: 5488 Loss: 0.01687951758503914
Epoch: 5489 Loss: 0.01687893457710743
Epoch: 5490 Loss: 0.016868287697434425
Epoch: 5491 Loss: 0.016866326332092285
Epoch: 5492 Loss: 0.016860902309417725
Epoch: 5493 Loss: 0.016852829605340958
Epoch: 5494 Loss: 0.01685190573334694
Epoch: 5495 Loss: 0.016845833510160446
Epoch: 5496 Loss: 0.016846762970089912
Epoch: 5497 Loss: 0.016834912821650505
Epoch: 5498 Loss: 0.0168281439691782
Epoch: 5499 Loss: 0.016825811937451363
Epoch: 5500 Loss: 0.016822610050439835
Epoch: 5501 Loss: 0.016811855137348175
Epoch: 5502 Loss: 0.016809826716780663
Epoch: 5503 Loss: 0.016805322840809822
Epoch: 5504 Loss: 0.016801413148641586
Epoch: 5505 Loss: 0.016794001683592796
Epoch: 5506 Loss: 0.016790451481938362
Epoch: 5507 Loss: 0.016786180436611176
Epoch: 5508 Loss: 0.016777336597442627
Epoch: 5509 Loss: 0.016771821305155754
Epoch: 5510 Loss: 0.016766713932156563
Epoch: 5511 Loss: 0.016764376312494278
Epoch: 5512 Loss: 0.016759682446718216
Epoch: 5513 Loss: 0.016753090545535088
Epoch: 5514 Loss: 0.016752852126955986
Epoch: 5515 Loss: 0.016743889078497887
Epoch: 5516 Loss: 0.01673883944749832
Epoch: 5517 Loss: 0.016732964664697647
Epoch: 5518 Loss: 0.016727523878216743
Epoch: 5519 Loss: 0.016725506633520126
Epoch: 5520 Loss: 0.016718681901693344
Epoch: 5521 Loss: 0.01671598292887211
Epoch: 5522 Loss: 0.016708964481949806
Epoch: 5523 Loss: 0.016705863177776337
Epoch: 5524 Loss: 0.016701798886060715
Epoch: 5525 Loss: 0.016693538054823875
Epoch: 5526 Loss: 0.016687829047441483
Epoch: 5527 Loss: 0.016684183850884438
Epoch: 5528 Loss: 0.016680100932717323
Epoch: 5529 Loss: 0.01667146012187004
Epoch: 5530 Loss: 0.01666676253080368
Epoch: 5531 Loss: 0.01666201837360859
Epoch: 5532 Loss: 0.016662094742059708
Epoch: 5533 Loss: 0.016656484454870224
Epoch: 5534 Loss: 0.01665167510509491
Epoch: 5535 Loss: 0.016643686220049858
Epoch: 5536 Loss: 0.016638578847050667
Epoch: 5537 Loss: 0.016634009778499603
Epoch: 5538 Loss: 0.016634222120046616
Epoch: 5539 Loss: 0.016621796414256096
Epoch: 5540 Loss: 0.01661662757396698
Epoch: 5541 Loss: 0.01661420799791813
Epoch: 5542 Loss: 0.016610093414783478
Epoch: 5543 Loss: 0.01660676673054695
Epoch: 5544 Loss: 0.016598382964730263
Epoch: 5545 Loss: 0.01659635454416275
Epoch: 5546 Loss: 0.016590995714068413
Epoch: 5547 Loss: 0.01658513769507408
Epoch: 5548 Loss: 0.016578244045376778
Epoch: 5549 Loss: 0.01657295599579811
Epoch: 5550 Loss: 0.016570108011364937
Epoch: 5551 Loss: 0.016565663740038872
Epoch: 5552 Loss: 0.016558369621634483
Epoch: 5553 Loss: 0.01655367761850357
Epoch: 5554 Loss: 0.01654866337776184
Epoch: 5555 Loss: 0.016543885692954063
Epoch: 5556 Loss: 0.016543306410312653
Epoch: 5557 Loss: 0.01653359830379486
Epoch: 5558 Loss: 0.016529520973563194
Epoch: 5559 Loss: 0.0165259912610054
Epoch: 5560 Loss: 0.016517648473381996
Epoch: 5561 Loss: 0.016514714807271957
Epoch: 5562 Loss: 0.016511626541614532
Epoch: 5563 Loss: 0.01650717481970787
Epoch: 5564 Loss: 0.016503755003213882
Epoch: 5565 Loss: 0.016492810100317
Epoch: 5566 Loss: 0.01649196445941925
Epoch: 5567 Loss: 0.016485000029206276
Epoch: 5568 Loss: 0.016482962295413017
Epoch: 5569 Loss: 0.016475938260555267
Epoch: 5570 Loss: 0.016472212970256805
Epoch: 5571 Loss: 0.01646418124437332
Epoch: 5572 Loss: 0.01646220125257969
Epoch: 5573 Loss: 0.016459140926599503
Epoch: 5574 Loss: 0.01645032688975334
Epoch: 5575 Loss: 0.016445543617010117
Epoch: 5576 Loss: 0.016444094479084015
Epoch: 5577 Loss: 0.01643618941307068
Epoch: 5578 Loss: 0.016428936272859573
Epoch: 5579 Loss: 0.016425061970949173
Epoch: 5580 Loss: 0.016422197222709656
Epoch: 5581 Loss: 0.016416560858488083
Epoch: 5582 Loss: 0.016411440446972847
Epoch: 5583 Loss: 0.01640971191227436
Epoch: 5584 Loss: 0.016399795189499855
Epoch: 5585 Loss: 0.01639954000711441
Epoch: 5586 Loss: 0.016392704099416733
Epoch: 5587 Loss: 0.016386285424232483
Epoch: 5588 Loss: 0.01638197898864746
Epoch: 5589 Loss: 0.016379836946725845
Epoch: 5590 Loss: 0.016375329345464706
Epoch: 5591 Loss: 0.016368601471185684
Epoch: 5592 Loss: 0.01636444590985775
Epoch: 5593 Loss: 0.016357973217964172
Epoch: 5594 Loss: 0.016350844874978065
Epoch: 5595 Loss: 0.01634977012872696
Epoch: 5596 Loss: 0.016342643648386
Epoch: 5597 Loss: 0.016339011490345
Epoch: 5598 Loss: 0.016336295753717422
Epoch: 5599 Loss: 0.016332168132066727
Epoch: 5600 Loss: 0.016328446567058563
Epoch: 5601 Loss: 0.01631922647356987
Epoch: 5602 Loss: 0.016315918415784836
Epoch: 5603 Loss: 0.01631004549562931
Epoch: 5604 Loss: 0.016306303441524506
Epoch: 5605 Loss: 0.016301188617944717
Epoch: 5606 Loss: 0.016294684261083603
Epoch: 5607 Loss: 0.01629027910530567
Epoch: 5608 Loss: 0.016286909580230713
Epoch: 5609 Loss: 0.016281574964523315
Epoch: 5610 Loss: 0.01627654954791069
Epoch: 5611 Loss: 0.016270870342850685
Epoch: 5612 Loss: 0.0162684116512537
Epoch: 5613 Loss: 0.016260895878076553
Epoch: 5614 Loss: 0.016256937757134438
Epoch: 5615 Loss: 0.016253888607025146
Epoch: 5616 Loss: 0.016248926520347595
Epoch: 5617 Loss: 0.01624729298055172
Epoch: 5618 Loss: 0.016243385151028633
Epoch: 5619 Loss: 0.01623210683465004
Epoch: 5620 Loss: 0.0162286926060915
Epoch: 5621 Loss: 0.016223348677158356
Epoch: 5622 Loss: 0.016216374933719635
Epoch: 5623 Loss: 0.016213933005928993
Epoch: 5624 Loss: 0.016213783994317055
Epoch: 5625 Loss: 0.016202392056584358
Epoch: 5626 Loss: 0.016199186444282532
Epoch: 5627 Loss: 0.016193009912967682
Epoch: 5628 Loss: 0.016194254159927368
Epoch: 5629 Loss: 0.016183698549866676
Epoch: 5630 Loss: 0.016180550679564476
Epoch: 5631 Loss: 0.01617564633488655
Epoch: 5632 Loss: 0.016172684729099274
Epoch: 5633 Loss: 0.01617266796529293
Epoch: 5634 Loss: 0.016161642968654633
Epoch: 5635 Loss: 0.016154969111084938
Epoch: 5636 Loss: 0.01615525595843792
Epoch: 5637 Loss: 0.016147170215845108
Epoch: 5638 Loss: 0.01614103838801384
Epoch: 5639 Loss: 0.0161378625780344
Epoch: 5640 Loss: 0.01613287627696991
Epoch: 5641 Loss: 0.016129279509186745
Epoch: 5642 Loss: 0.016124868765473366
Epoch: 5643 Loss: 0.01612476073205471
Epoch: 5644 Loss: 0.01611246168613434
Epoch: 5645 Loss: 0.016106557101011276
Epoch: 5646 Loss: 0.016105590388178825
Epoch: 5647 Loss: 0.01610046997666359
Epoch: 5648 Loss: 0.01610175333917141
Epoch: 5649 Loss: 0.01609010063111782
Epoch: 5650 Loss: 0.01608671247959137
Epoch: 5651 Loss: 0.016081996262073517
Epoch: 5652 Loss: 0.016078531742095947
Epoch: 5653 Loss: 0.01607241854071617
Epoch: 5654 Loss: 0.01606777496635914
Epoch: 5655 Loss: 0.016063567250967026
Epoch: 5656 Loss: 0.016060907393693924
Epoch: 5657 Loss: 0.016053801402449608
Epoch: 5658 Loss: 0.01604921743273735
Epoch: 5659 Loss: 0.01604480668902397
Epoch: 5660 Loss: 0.016038015484809875
Epoch: 5661 Loss: 0.016034850850701332
Epoch: 5662 Loss: 0.016029708087444305
Epoch: 5663 Loss: 0.01602783426642418
Epoch: 5664 Loss: 0.01602116785943508
Epoch: 5665 Loss: 0.016018034890294075
Epoch: 5666 Loss: 0.016011759638786316
Epoch: 5667 Loss: 0.016007225960493088
Epoch: 5668 Loss: 0.016001049429178238
Epoch: 5669 Loss: 0.0159975066781044
Epoch: 5670 Loss: 0.01599210873246193
Epoch: 5671 Loss: 0.01599162071943283
Epoch: 5672 Loss: 0.01598251610994339
Epoch: 5673 Loss: 0.015976935625076294
Epoch: 5674 Loss: 0.015972673892974854
Epoch: 5675 Loss: 0.015974169597029686
Epoch: 5676 Loss: 0.015963347628712654
Epoch: 5677 Loss: 0.015959978103637695
Epoch: 5678 Loss: 0.01595480926334858
Epoch: 5679 Loss: 0.01595252752304077
Epoch: 5680 Loss: 0.01594715379178524
Epoch: 5681 Loss: 0.015940841287374496
Epoch: 5682 Loss: 0.01593722403049469
Epoch: 5683 Loss: 0.015934690833091736
Epoch: 5684 Loss: 0.01593117229640484
Epoch: 5685 Loss: 0.01592111773788929
Epoch: 5686 Loss: 0.015920741483569145
Epoch: 5687 Loss: 0.01591232605278492
Epoch: 5688 Loss: 0.015909358859062195
Epoch: 5689 Loss: 0.015904942527413368
Epoch: 5690 Loss: 0.01590108498930931
Epoch: 5691 Loss: 0.01589498296380043
Epoch: 5692 Loss: 0.01589171774685383
Epoch: 5693 Loss: 0.015888018533587456
Epoch: 5694 Loss: 0.015879273414611816
Epoch: 5695 Loss: 0.015878180041909218
Epoch: 5696 Loss: 0.0158736240118742
Epoch: 5697 Loss: 0.01586895063519478
Epoch: 5698 Loss: 0.015865681692957878
Epoch: 5699 Loss: 0.015858596190810204
Epoch: 5700 Loss: 0.015854328870773315
Epoch: 5701 Loss: 0.01585102640092373
Epoch: 5702 Loss: 0.015843801200389862
Epoch: 5703 Loss: 0.015840543434023857
Epoch: 5704 Loss: 0.01583615504205227
Epoch: 5705 Loss: 0.015833357349038124
Epoch: 5706 Loss: 0.015828439965844154
Epoch: 5707 Loss: 0.015822185203433037
Epoch: 5708 Loss: 0.015820743516087532
Epoch: 5709 Loss: 0.015812506899237633
Epoch: 5710 Loss: 0.01580755226314068
Epoch: 5711 Loss: 0.015806106850504875
Epoch: 5712 Loss: 0.01579849235713482
Epoch: 5713 Loss: 0.015795094892382622
Epoch: 5714 Loss: 0.015789812430739403
Epoch: 5715 Loss: 0.015785450115799904
Epoch: 5716 Loss: 0.015781158581376076
Epoch: 5717 Loss: 0.015774991363286972
Epoch: 5718 Loss: 0.015774225816130638
Epoch: 5719 Loss: 0.015768004581332207
Epoch: 5720 Loss: 0.015764867886900902
Epoch: 5721 Loss: 0.015757421031594276
Epoch: 5722 Loss: 0.01575520634651184
Epoch: 5723 Loss: 0.015753060579299927
Epoch: 5724 Loss: 0.015745509415864944
Epoch: 5725 Loss: 0.015738695859909058
Epoch: 5726 Loss: 0.015736956149339676
Epoch: 5727 Loss: 0.015733972191810608
Epoch: 5728 Loss: 0.015727879479527473
Epoch: 5729 Loss: 0.015722433105111122
Epoch: 5730 Loss: 0.015715761110186577
Epoch: 5731 Loss: 0.015714148059487343
Epoch: 5732 Loss: 0.015708589926362038
Epoch: 5733 Loss: 0.015704253688454628
Epoch: 5734 Loss: 0.015698663890361786
Epoch: 5735 Loss: 0.015695661306381226
Epoch: 5736 Loss: 0.01569364406168461
Epoch: 5737 Loss: 0.015686459839344025
Epoch: 5738 Loss: 0.015682019293308258
Epoch: 5739 Loss: 0.01567664183676243
Epoch: 5740 Loss: 0.015676230192184448
Epoch: 5741 Loss: 0.015667051076889038
Epoch: 5742 Loss: 0.01566389948129654
Epoch: 5743 Loss: 0.015657998621463776
Epoch: 5744 Loss: 0.01565994881093502
Epoch: 5745 Loss: 0.01564958319067955
Epoch: 5746 Loss: 0.01564423367381096
Epoch: 5747 Loss: 0.015643663704395294
Epoch: 5748 Loss: 0.015637846663594246
Epoch: 5749 Loss: 0.015630140900611877
Epoch: 5750 Loss: 0.015627415850758553
Epoch: 5751 Loss: 0.015622299164533615
Epoch: 5752 Loss: 0.015618427656590939
Epoch: 5753 Loss: 0.015617107041180134
Epoch: 5754 Loss: 0.015610706992447376
Epoch: 5755 Loss: 0.015607352368533611
Epoch: 5756 Loss: 0.015603099949657917
Epoch: 5757 Loss: 0.015598537400364876
Epoch: 5758 Loss: 0.015594305470585823
Epoch: 5759 Loss: 0.01558744627982378
Epoch: 5760 Loss: 0.015581667423248291
Epoch: 5761 Loss: 0.015579327009618282
Epoch: 5762 Loss: 0.015575631521642208
Epoch: 5763 Loss: 0.015569073148071766
Epoch: 5764 Loss: 0.015565069392323494
Epoch: 5765 Loss: 0.01556042768061161
Epoch: 5766 Loss: 0.015556997619569302
Epoch: 5767 Loss: 0.015554716810584068
Epoch: 5768 Loss: 0.015547594055533409
Epoch: 5769 Loss: 0.015543946996331215
Epoch: 5770 Loss: 0.015538020990788937
Epoch: 5771 Loss: 0.015537794679403305
Epoch: 5772 Loss: 0.015528730116784573
Epoch: 5773 Loss: 0.015523928217589855
Epoch: 5774 Loss: 0.01552241574972868
Epoch: 5775 Loss: 0.015518320724368095
Epoch: 5776 Loss: 0.015510909259319305
Epoch: 5777 Loss: 0.015508657321333885
Epoch: 5778 Loss: 0.01550679188221693
Epoch: 5779 Loss: 0.015500095672905445
Epoch: 5780 Loss: 0.015495176427066326
Epoch: 5781 Loss: 0.015491428785026073
Epoch: 5782 Loss: 0.015483811497688293
Epoch: 5783 Loss: 0.015483210794627666
Epoch: 5784 Loss: 0.015474970452487469
Epoch: 5785 Loss: 0.015474938787519932
Epoch: 5786 Loss: 0.015467868186533451
Epoch: 5787 Loss: 0.015463476069271564
Epoch: 5788 Loss: 0.015459815040230751
Epoch: 5789 Loss: 0.015454702079296112
Epoch: 5790 Loss: 0.015449169091880322
Epoch: 5791 Loss: 0.015448485501110554
Epoch: 5792 Loss: 0.015443865209817886
Epoch: 5793 Loss: 0.01543607097119093
Epoch: 5794 Loss: 0.015434522181749344
Epoch: 5795 Loss: 0.015430494211614132
Epoch: 5796 Loss: 0.015428205952048302
Epoch: 5797 Loss: 0.015417148359119892
Epoch: 5798 Loss: 0.015414658933877945
Epoch: 5799 Loss: 0.015412637032568455
Epoch: 5800 Loss: 0.015410449355840683
Epoch: 5801 Loss: 0.015402992255985737
Epoch: 5802 Loss: 0.015399656258523464
Epoch: 5803 Loss: 0.015394877642393112
Epoch: 5804 Loss: 0.01539135817438364
Epoch: 5805 Loss: 0.015388507395982742
Epoch: 5806 Loss: 0.015380683355033398
Epoch: 5807 Loss: 0.015377654694020748
Epoch: 5808 Loss: 0.015373768284916878
Epoch: 5809 Loss: 0.015367713756859303
Epoch: 5810 Loss: 0.015364698134362698
Epoch: 5811 Loss: 0.015358041040599346
Epoch: 5812 Loss: 0.015353193506598473
Epoch: 5813 Loss: 0.015349453315138817
Epoch: 5814 Loss: 0.015346172265708447
Epoch: 5815 Loss: 0.015344580635428429
Epoch: 5816 Loss: 0.01533756498247385
Epoch: 5817 Loss: 0.015333184041082859
Epoch: 5818 Loss: 0.015332715585827827
Epoch: 5819 Loss: 0.015324754640460014
Epoch: 5820 Loss: 0.015322024002671242
Epoch: 5821 Loss: 0.0153184300288558
Epoch: 5822 Loss: 0.01531161554157734
Epoch: 5823 Loss: 0.015307066030800343
Epoch: 5824 Loss: 0.015301734209060669
Epoch: 5825 Loss: 0.015300230123102665
Epoch: 5826 Loss: 0.015296678058803082
Epoch: 5827 Loss: 0.015290960669517517
Epoch: 5828 Loss: 0.015285876579582691
Epoch: 5829 Loss: 0.015282610431313515
Epoch: 5830 Loss: 0.015277489088475704
Epoch: 5831 Loss: 0.015274074859917164
Epoch: 5832 Loss: 0.015267914161086082
Epoch: 5833 Loss: 0.015265076421201229
Epoch: 5834 Loss: 0.015261932276189327
Epoch: 5835 Loss: 0.015254944562911987
Epoch: 5836 Loss: 0.01525044534355402
Epoch: 5837 Loss: 0.015247020870447159
Epoch: 5838 Loss: 0.015244673006236553
Epoch: 5839 Loss: 0.015237324871122837
Epoch: 5840 Loss: 0.015234898775815964
Epoch: 5841 Loss: 0.015228679403662682
Epoch: 5842 Loss: 0.015226823277771473
Epoch: 5843 Loss: 0.015220562927424908
Epoch: 5844 Loss: 0.015217816457152367
Epoch: 5845 Loss: 0.01521332748234272
Epoch: 5846 Loss: 0.01521284319460392
Epoch: 5847 Loss: 0.015204858034849167
Epoch: 5848 Loss: 0.015200200490653515
Epoch: 5849 Loss: 0.01519633550196886
Epoch: 5850 Loss: 0.015194281004369259
Epoch: 5851 Loss: 0.015186024829745293
Epoch: 5852 Loss: 0.01518163364380598
Epoch: 5853 Loss: 0.01517876610159874
Epoch: 5854 Loss: 0.015175435692071915
Epoch: 5855 Loss: 0.015170828439295292
Epoch: 5856 Loss: 0.015170465223491192
Epoch: 5857 Loss: 0.015162884257733822
Epoch: 5858 Loss: 0.015157786197960377
Epoch: 5859 Loss: 0.015152453444898129
Epoch: 5860 Loss: 0.015148760750889778
Epoch: 5861 Loss: 0.015146804973483086
Epoch: 5862 Loss: 0.015143591910600662
Epoch: 5863 Loss: 0.015138356015086174
Epoch: 5864 Loss: 0.015130157582461834
Epoch: 5865 Loss: 0.015126893296837807
Epoch: 5866 Loss: 0.015126040205359459
Epoch: 5867 Loss: 0.015123018994927406
Epoch: 5868 Loss: 0.015117645263671875
Epoch: 5869 Loss: 0.015109555795788765
Epoch: 5870 Loss: 0.015105675905942917
Epoch: 5871 Loss: 0.015105867758393288
Epoch: 5872 Loss: 0.01510111428797245
Epoch: 5873 Loss: 0.015095589682459831
Epoch: 5874 Loss: 0.015090511180460453
Epoch: 5875 Loss: 0.01508727204054594
Epoch: 5876 Loss: 0.015081454068422318
Epoch: 5877 Loss: 0.015078241005539894
Epoch: 5878 Loss: 0.015073815360665321
Epoch: 5879 Loss: 0.015071801841259003
Epoch: 5880 Loss: 0.01506540272384882
Epoch: 5881 Loss: 0.015059049241244793
Epoch: 5882 Loss: 0.015059619210660458
Epoch: 5883 Loss: 0.015053918585181236
Epoch: 5884 Loss: 0.015046891756355762
Epoch: 5885 Loss: 0.015044468455016613
Epoch: 5886 Loss: 0.015039367601275444
Epoch: 5887 Loss: 0.015035015530884266
Epoch: 5888 Loss: 0.0150308171287179
Epoch: 5889 Loss: 0.015026857145130634
Epoch: 5890 Loss: 0.015023268759250641
Epoch: 5891 Loss: 0.01502173114567995
Epoch: 5892 Loss: 0.015017934143543243
Epoch: 5893 Loss: 0.015011927112936974
Epoch: 5894 Loss: 0.01500645000487566
Epoch: 5895 Loss: 0.015003121457993984
Epoch: 5896 Loss: 0.014998428523540497
Epoch: 5897 Loss: 0.014993610791862011
Epoch: 5898 Loss: 0.014990692026913166
Epoch: 5899 Loss: 0.014985727146267891
Epoch: 5900 Loss: 0.014980712905526161
Epoch: 5901 Loss: 0.014977597631514072
Epoch: 5902 Loss: 0.014976960606873035
Epoch: 5903 Loss: 0.014967761933803558
Epoch: 5904 Loss: 0.01496448740363121
Epoch: 5905 Loss: 0.014962091110646725
Epoch: 5906 Loss: 0.014956693165004253
Epoch: 5907 Loss: 0.014952008612453938
Epoch: 5908 Loss: 0.014951334334909916
Epoch: 5909 Loss: 0.014942430891096592
Epoch: 5910 Loss: 0.014943428337574005
Epoch: 5911 Loss: 0.014936573803424835
Epoch: 5912 Loss: 0.014934791252017021
Epoch: 5913 Loss: 0.014929317869246006
Epoch: 5914 Loss: 0.014924127608537674
Epoch: 5915 Loss: 0.014921402558684349
Epoch: 5916 Loss: 0.014916922897100449
Epoch: 5917 Loss: 0.01491136197000742
Epoch: 5918 Loss: 0.014906871132552624
Epoch: 5919 Loss: 0.014904393814504147
Epoch: 5920 Loss: 0.014900102280080318
Epoch: 5921 Loss: 0.01489434763789177
Epoch: 5922 Loss: 0.014891423285007477
Epoch: 5923 Loss: 0.014888538047671318
Epoch: 5924 Loss: 0.014884025789797306
Epoch: 5925 Loss: 0.014878175221383572
Epoch: 5926 Loss: 0.014874355867505074
Epoch: 5927 Loss: 0.014871429651975632
Epoch: 5928 Loss: 0.014868228696286678
Epoch: 5929 Loss: 0.014861664734780788
Epoch: 5930 Loss: 0.014857548289000988
Epoch: 5931 Loss: 0.014854129403829575
Epoch: 5932 Loss: 0.014853731729090214
Epoch: 5933 Loss: 0.014845221303403378
Epoch: 5934 Loss: 0.014839951880276203
Epoch: 5935 Loss: 0.014838133938610554
Epoch: 5936 Loss: 0.014833512715995312
Epoch: 5937 Loss: 0.01483030617237091
Epoch: 5938 Loss: 0.014830010943114758
Epoch: 5939 Loss: 0.014820504933595657
Epoch: 5940 Loss: 0.01481679268181324
Epoch: 5941 Loss: 0.01481341291218996
Epoch: 5942 Loss: 0.014811196364462376
Epoch: 5943 Loss: 0.01480390690267086
Epoch: 5944 Loss: 0.01480233483016491
Epoch: 5945 Loss: 0.014797700569033623
Epoch: 5946 Loss: 0.014792432077229023
Epoch: 5947 Loss: 0.01478959433734417
Epoch: 5948 Loss: 0.014787268824875355
Epoch: 5949 Loss: 0.014781700447201729
Epoch: 5950 Loss: 0.014777926728129387
Epoch: 5951 Loss: 0.014772896654903889
Epoch: 5952 Loss: 0.014770430512726307
Epoch: 5953 Loss: 0.014765496365725994
Epoch: 5954 Loss: 0.014758720062673092
Epoch: 5955 Loss: 0.014758137986063957
Epoch: 5956 Loss: 0.014751662500202656
Epoch: 5957 Loss: 0.014748262241482735
Epoch: 5958 Loss: 0.014746468514204025
Epoch: 5959 Loss: 0.014742165803909302
Epoch: 5960 Loss: 0.01473938673734665
Epoch: 5961 Loss: 0.014732519164681435
Epoch: 5962 Loss: 0.014728216454386711
Epoch: 5963 Loss: 0.014724969863891602
Epoch: 5964 Loss: 0.014719982631504536
Epoch: 5965 Loss: 0.014716175384819508
Epoch: 5966 Loss: 0.014716309495270252
Epoch: 5967 Loss: 0.014710400253534317
Epoch: 5968 Loss: 0.014704828150570393
Epoch: 5969 Loss: 0.014699573628604412
Epoch: 5970 Loss: 0.014699009247124195
Epoch: 5971 Loss: 0.014692101627588272
Epoch: 5972 Loss: 0.014686515554785728
Epoch: 5973 Loss: 0.01468590460717678
Epoch: 5974 Loss: 0.014679040759801865
Epoch: 5975 Loss: 0.014675826765596867
Epoch: 5976 Loss: 0.014672295190393925
Epoch: 5977 Loss: 0.014671092852950096
Epoch: 5978 Loss: 0.01466696709394455
Epoch: 5979 Loss: 0.014661962166428566
Epoch: 5980 Loss: 0.014656285755336285
Epoch: 5981 Loss: 0.014651707373559475
Epoch: 5982 Loss: 0.014650629833340645
Epoch: 5983 Loss: 0.014646182768046856
Epoch: 5984 Loss: 0.01464121788740158
Epoch: 5985 Loss: 0.01463859062641859
Epoch: 5986 Loss: 0.014631017111241817
Epoch: 5987 Loss: 0.014628463424742222
Epoch: 5988 Loss: 0.014624443836510181
Epoch: 5989 Loss: 0.014621265232563019
Epoch: 5990 Loss: 0.01461612805724144
Epoch: 5991 Loss: 0.014615275897085667
Epoch: 5992 Loss: 0.014610451646149158
Epoch: 5993 Loss: 0.014605420641601086
Epoch: 5994 Loss: 0.014600240625441074
Epoch: 5995 Loss: 0.01459699496626854
Epoch: 5996 Loss: 0.014594444073736668
Epoch: 5997 Loss: 0.014588872902095318
Epoch: 5998 Loss: 0.014582952484488487
Epoch: 5999 Loss: 0.014581491239368916
Epoch: 6000 Loss: 0.014578305184841156
Epoch: 6001 Loss: 0.014573225751519203
Epoch: 6002 Loss: 0.014569547027349472
Epoch: 6003 Loss: 0.014568569138646126
Epoch: 6004 Loss: 0.014562190510332584
Epoch: 6005 Loss: 0.014559458941221237
Epoch: 6006 Loss: 0.014553128741681576
Epoch: 6007 Loss: 0.01454913429915905
Epoch: 6008 Loss: 0.014543751254677773
Epoch: 6009 Loss: 0.014544472098350525
Epoch: 6010 Loss: 0.01454021967947483
Epoch: 6011 Loss: 0.014535239897668362
Epoch: 6012 Loss: 0.014530472457408905
Epoch: 6013 Loss: 0.014526821672916412
Epoch: 6014 Loss: 0.014521535485982895
Epoch: 6015 Loss: 0.014518464915454388
Epoch: 6016 Loss: 0.014517126604914665
Epoch: 6017 Loss: 0.01450975239276886
Epoch: 6018 Loss: 0.014509732834994793
Epoch: 6019 Loss: 0.014504341408610344
Epoch: 6020 Loss: 0.014497560448944569
Epoch: 6021 Loss: 0.014495985582470894
Epoch: 6022 Loss: 0.014491595327854156
Epoch: 6023 Loss: 0.014487863518297672
Epoch: 6024 Loss: 0.014482865110039711
Epoch: 6025 Loss: 0.014477013610303402
Epoch: 6026 Loss: 0.014474345371127129
Epoch: 6027 Loss: 0.01447056233882904
Epoch: 6028 Loss: 0.014467013068497181
Epoch: 6029 Loss: 0.014466093853116035
Epoch: 6030 Loss: 0.014460334554314613
Epoch: 6031 Loss: 0.014456144534051418
Epoch: 6032 Loss: 0.014450346119701862
Epoch: 6033 Loss: 0.014449093490839005
Epoch: 6034 Loss: 0.014444022439420223
Epoch: 6035 Loss: 0.014438449405133724
Epoch: 6036 Loss: 0.014438631944358349
Epoch: 6037 Loss: 0.014432655647397041
Epoch: 6038 Loss: 0.01443069614470005
Epoch: 6039 Loss: 0.014423995278775692
Epoch: 6040 Loss: 0.014422833919525146
Epoch: 6041 Loss: 0.014418908394873142
Epoch: 6042 Loss: 0.014412066899240017
Epoch: 6043 Loss: 0.014408848248422146
Epoch: 6044 Loss: 0.014405147172510624
Epoch: 6045 Loss: 0.014404045417904854
Epoch: 6046 Loss: 0.01439872570335865
Epoch: 6047 Loss: 0.014391938224434853
Epoch: 6048 Loss: 0.014389365911483765
Epoch: 6049 Loss: 0.014385571703314781
Epoch: 6050 Loss: 0.014383419416844845
Epoch: 6051 Loss: 0.014379416592419147
Epoch: 6052 Loss: 0.014373012818396091
Epoch: 6053 Loss: 0.014372438192367554
Epoch: 6054 Loss: 0.014368273317813873
Epoch: 6055 Loss: 0.014364894479513168
Epoch: 6056 Loss: 0.014357220381498337
Epoch: 6057 Loss: 0.014358180575072765
Epoch: 6058 Loss: 0.014349070377647877
Epoch: 6059 Loss: 0.014346812851727009
Epoch: 6060 Loss: 0.014344313181936741
Epoch: 6061 Loss: 0.014340220019221306
Epoch: 6062 Loss: 0.014334283769130707
Epoch: 6063 Loss: 0.014332669787108898
Epoch: 6064 Loss: 0.01432888861745596
Epoch: 6065 Loss: 0.014326607808470726
Epoch: 6066 Loss: 0.014320786111056805
Epoch: 6067 Loss: 0.014315769076347351
Epoch: 6068 Loss: 0.014313047751784325
Epoch: 6069 Loss: 0.014309369958937168
Epoch: 6070 Loss: 0.014305364340543747
Epoch: 6071 Loss: 0.014302306808531284
Epoch: 6072 Loss: 0.01429842971265316
Epoch: 6073 Loss: 0.014294670894742012
Epoch: 6074 Loss: 0.014290105551481247
Epoch: 6075 Loss: 0.014286601915955544
Epoch: 6076 Loss: 0.014284027740359306
Epoch: 6077 Loss: 0.014281262643635273
Epoch: 6078 Loss: 0.014277092181146145
Epoch: 6079 Loss: 0.014272044412791729
Epoch: 6080 Loss: 0.01426601316779852
Epoch: 6081 Loss: 0.014262020587921143
Epoch: 6082 Loss: 0.014259102754294872
Epoch: 6083 Loss: 0.014255011454224586
Epoch: 6084 Loss: 0.014252811670303345
Epoch: 6085 Loss: 0.014249997213482857
Epoch: 6086 Loss: 0.014243739657104015
Epoch: 6087 Loss: 0.014240800403058529
Epoch: 6088 Loss: 0.0142386294901371
Epoch: 6089 Loss: 0.014233788475394249
Epoch: 6090 Loss: 0.014232126995921135
Epoch: 6091 Loss: 0.014226638711988926
Epoch: 6092 Loss: 0.01422097347676754
Epoch: 6093 Loss: 0.014218279160559177
Epoch: 6094 Loss: 0.0142155010253191
Epoch: 6095 Loss: 0.014212927781045437
Epoch: 6096 Loss: 0.01420612446963787
Epoch: 6097 Loss: 0.014202145859599113
Epoch: 6098 Loss: 0.014198873192071915
Epoch: 6099 Loss: 0.014196817763149738
Epoch: 6100 Loss: 0.014191882684826851
Epoch: 6101 Loss: 0.014187516644597054
Epoch: 6102 Loss: 0.014186411164700985
Epoch: 6103 Loss: 0.014180795289576054
Epoch: 6104 Loss: 0.014175165444612503
Epoch: 6105 Loss: 0.014174363575875759
Epoch: 6106 Loss: 0.014171481132507324
Epoch: 6107 Loss: 0.01416792068630457
Epoch: 6108 Loss: 0.014162820763885975
Epoch: 6109 Loss: 0.014159204438328743
Epoch: 6110 Loss: 0.014152838848531246
Epoch: 6111 Loss: 0.014151022769510746
Epoch: 6112 Loss: 0.014146538451313972
Epoch: 6113 Loss: 0.014143243432044983
Epoch: 6114 Loss: 0.014136535115540028
Epoch: 6115 Loss: 0.014136883430182934
Epoch: 6116 Loss: 0.014131869189441204
Epoch: 6117 Loss: 0.014126941561698914
Epoch: 6118 Loss: 0.014123431406915188
Epoch: 6119 Loss: 0.014120528474450111
Epoch: 6120 Loss: 0.01412037294358015
Epoch: 6121 Loss: 0.014115633442997932
Epoch: 6122 Loss: 0.014108389616012573
Epoch: 6123 Loss: 0.01410672813653946
Epoch: 6124 Loss: 0.01410050131380558
Epoch: 6125 Loss: 0.01409931294620037
Epoch: 6126 Loss: 0.014094753190875053
Epoch: 6127 Loss: 0.0140922786667943
Epoch: 6128 Loss: 0.014090740121901035
Epoch: 6129 Loss: 0.014083506539463997
Epoch: 6130 Loss: 0.01408185251057148
Epoch: 6131 Loss: 0.014077495783567429
Epoch: 6132 Loss: 0.014073316007852554
Epoch: 6133 Loss: 0.014069226570427418
Epoch: 6134 Loss: 0.014064171351492405
Epoch: 6135 Loss: 0.014063948765397072
Epoch: 6136 Loss: 0.01405669841915369
Epoch: 6137 Loss: 0.014053238555788994
Epoch: 6138 Loss: 0.014051387086510658
Epoch: 6139 Loss: 0.014046275988221169
Epoch: 6140 Loss: 0.014046921394765377
Epoch: 6141 Loss: 0.014040648005902767
Epoch: 6142 Loss: 0.014036227948963642
Epoch: 6143 Loss: 0.014033166691660881
Epoch: 6144 Loss: 0.01402647327631712
Epoch: 6145 Loss: 0.014024992473423481
Epoch: 6146 Loss: 0.014021419920027256
Epoch: 6147 Loss: 0.014018352143466473
Epoch: 6148 Loss: 0.01401490904390812
Epoch: 6149 Loss: 0.014009534381330013
Epoch: 6150 Loss: 0.01400536298751831
Epoch: 6151 Loss: 0.014002924785017967
Epoch: 6152 Loss: 0.013998000882565975
Epoch: 6153 Loss: 0.013993969187140465
Epoch: 6154 Loss: 0.01399009395390749
Epoch: 6155 Loss: 0.013990379869937897
Epoch: 6156 Loss: 0.013983002863824368
Epoch: 6157 Loss: 0.013979523442685604
Epoch: 6158 Loss: 0.0139764454215765
Epoch: 6159 Loss: 0.013973924331367016
Epoch: 6160 Loss: 0.01396974641829729
Epoch: 6161 Loss: 0.013966638594865799
Epoch: 6162 Loss: 0.013961314223706722
Epoch: 6163 Loss: 0.013959307223558426
Epoch: 6164 Loss: 0.013955670408904552
Epoch: 6165 Loss: 0.013952149078249931
Epoch: 6166 Loss: 0.013950596563518047
Epoch: 6167 Loss: 0.013941489160060883
Epoch: 6168 Loss: 0.01393985003232956
Epoch: 6169 Loss: 0.013936513103544712
Epoch: 6170 Loss: 0.013933414593338966
Epoch: 6171 Loss: 0.013933461159467697
Epoch: 6172 Loss: 0.013927089050412178
Epoch: 6173 Loss: 0.013922015205025673
Epoch: 6174 Loss: 0.013919303193688393
Epoch: 6175 Loss: 0.013916040770709515
Epoch: 6176 Loss: 0.013914152979850769
Epoch: 6177 Loss: 0.013909821398556232
Epoch: 6178 Loss: 0.013902917504310608
Epoch: 6179 Loss: 0.013901411555707455
Epoch: 6180 Loss: 0.013897367753088474
Epoch: 6181 Loss: 0.013894417323172092
Epoch: 6182 Loss: 0.013888577930629253
Epoch: 6183 Loss: 0.013884680345654488
Epoch: 6184 Loss: 0.013882070779800415
Epoch: 6185 Loss: 0.013878446072340012
Epoch: 6186 Loss: 0.013875368982553482
Epoch: 6187 Loss: 0.01387039851397276
Epoch: 6188 Loss: 0.013870874419808388
Epoch: 6189 Loss: 0.013864164240658283
Epoch: 6190 Loss: 0.013859317637979984
Epoch: 6191 Loss: 0.013857051730155945
Epoch: 6192 Loss: 0.013853896409273148
Epoch: 6193 Loss: 0.01385032944381237
Epoch: 6194 Loss: 0.013848789967596531
Epoch: 6195 Loss: 0.013844926841557026
Epoch: 6196 Loss: 0.013841424137353897
Epoch: 6197 Loss: 0.013834992423653603
Epoch: 6198 Loss: 0.013834255747497082
Epoch: 6199 Loss: 0.01382968109101057
Epoch: 6200 Loss: 0.013824119232594967
Epoch: 6201 Loss: 0.013820627704262733
Epoch: 6202 Loss: 0.013819051906466484
Epoch: 6203 Loss: 0.013815614394843578
Epoch: 6204 Loss: 0.01380920223891735
Epoch: 6205 Loss: 0.013807221315801144
Epoch: 6206 Loss: 0.013804792426526546
Epoch: 6207 Loss: 0.013799017295241356
Epoch: 6208 Loss: 0.013797231018543243
Epoch: 6209 Loss: 0.013795841485261917
Epoch: 6210 Loss: 0.01379113458096981
Epoch: 6211 Loss: 0.01378596294671297
Epoch: 6212 Loss: 0.01378222182393074
Epoch: 6213 Loss: 0.013779397122561932
Epoch: 6214 Loss: 0.013777077198028564
Epoch: 6215 Loss: 0.013771472498774529
Epoch: 6216 Loss: 0.013767450116574764
Epoch: 6217 Loss: 0.013762839138507843
Epoch: 6218 Loss: 0.013761579990386963
Epoch: 6219 Loss: 0.013759241439402103
Epoch: 6220 Loss: 0.01375330239534378
Epoch: 6221 Loss: 0.01374922227114439
Epoch: 6222 Loss: 0.013747023418545723
Epoch: 6223 Loss: 0.013745034113526344
Epoch: 6224 Loss: 0.013739475980401039
Epoch: 6225 Loss: 0.01373547688126564
Epoch: 6226 Loss: 0.013731316663324833
Epoch: 6227 Loss: 0.013728725723922253
Epoch: 6228 Loss: 0.013730457052588463
Epoch: 6229 Loss: 0.0137226153165102
Epoch: 6230 Loss: 0.013718386180698872
Epoch: 6231 Loss: 0.013715664856135845
Epoch: 6232 Loss: 0.013714183121919632
Epoch: 6233 Loss: 0.013708489947021008
Epoch: 6234 Loss: 0.013705244287848473
Epoch: 6235 Loss: 0.013701004907488823
Epoch: 6236 Loss: 0.013699976727366447
Epoch: 6237 Loss: 0.013692723587155342
Epoch: 6238 Loss: 0.01369178481400013
Epoch: 6239 Loss: 0.013687106780707836
Epoch: 6240 Loss: 0.01368490420281887
Epoch: 6241 Loss: 0.013683546334505081
Epoch: 6242 Loss: 0.013680479489266872
Epoch: 6243 Loss: 0.013672485016286373
Epoch: 6244 Loss: 0.013669597916305065
Epoch: 6245 Loss: 0.013666883111000061
Epoch: 6246 Loss: 0.013664116151630878
Epoch: 6247 Loss: 0.013660789467394352
Epoch: 6248 Loss: 0.013655640184879303
Epoch: 6249 Loss: 0.01365165039896965
Epoch: 6250 Loss: 0.013651015236973763
Epoch: 6251 Loss: 0.013645092956721783
Epoch: 6252 Loss: 0.013642335310578346
Epoch: 6253 Loss: 0.01364106871187687
Epoch: 6254 Loss: 0.013635113835334778
Epoch: 6255 Loss: 0.013632386922836304
Epoch: 6256 Loss: 0.013630185276269913
Epoch: 6257 Loss: 0.013625077903270721
Epoch: 6258 Loss: 0.013621685095131397
Epoch: 6259 Loss: 0.013618165627121925
Epoch: 6260 Loss: 0.013616722077131271
Epoch: 6261 Loss: 0.013610871508717537
Epoch: 6262 Loss: 0.013606778346002102
Epoch: 6263 Loss: 0.01360238902270794
Epoch: 6264 Loss: 0.013603036291897297
Epoch: 6265 Loss: 0.013599526137113571
Epoch: 6266 Loss: 0.013594985008239746
Epoch: 6267 Loss: 0.013590841554105282
Epoch: 6268 Loss: 0.013586624525487423
Epoch: 6269 Loss: 0.013582178391516209
Epoch: 6270 Loss: 0.013579113408923149
Epoch: 6271 Loss: 0.013575712218880653
Epoch: 6272 Loss: 0.013574345968663692
Epoch: 6273 Loss: 0.013569356873631477
Epoch: 6274 Loss: 0.013565394096076488
Epoch: 6275 Loss: 0.013563641346991062
Epoch: 6276 Loss: 0.01356176845729351
Epoch: 6277 Loss: 0.013558275997638702
Epoch: 6278 Loss: 0.01355233695358038
Epoch: 6279 Loss: 0.013549074530601501
Epoch: 6280 Loss: 0.013545391149818897
Epoch: 6281 Loss: 0.013543762266635895
Epoch: 6282 Loss: 0.013538376428186893
Epoch: 6283 Loss: 0.013534699566662312
Epoch: 6284 Loss: 0.01353184413164854
Epoch: 6285 Loss: 0.01352657563984394
Epoch: 6286 Loss: 0.013524288311600685
Epoch: 6287 Loss: 0.013523228466510773
Epoch: 6288 Loss: 0.013521620072424412
Epoch: 6289 Loss: 0.013513986952602863
Epoch: 6290 Loss: 0.013512758538126945
Epoch: 6291 Loss: 0.013507617637515068
Epoch: 6292 Loss: 0.013503333553671837
Epoch: 6293 Loss: 0.013500736095011234
Epoch: 6294 Loss: 0.013499467633664608
Epoch: 6295 Loss: 0.013493283651769161
Epoch: 6296 Loss: 0.013490659184753895
Epoch: 6297 Loss: 0.013487685471773148
Epoch: 6298 Loss: 0.013483423739671707
Epoch: 6299 Loss: 0.013480518013238907
Epoch: 6300 Loss: 0.013479462824761868
Epoch: 6301 Loss: 0.013476084917783737
Epoch: 6302 Loss: 0.01347056683152914
Epoch: 6303 Loss: 0.013465258292853832
Epoch: 6304 Loss: 0.01346468087285757
Epoch: 6305 Loss: 0.013459783978760242
Epoch: 6306 Loss: 0.01345735602080822
Epoch: 6307 Loss: 0.013452541083097458
Epoch: 6308 Loss: 0.013451500795781612
Epoch: 6309 Loss: 0.013445510528981686
Epoch: 6310 Loss: 0.01344281155616045
Epoch: 6311 Loss: 0.013439701870083809
Epoch: 6312 Loss: 0.013436865992844105
Epoch: 6313 Loss: 0.013434227555990219
Epoch: 6314 Loss: 0.013431014493107796
Epoch: 6315 Loss: 0.013427725993096828
Epoch: 6316 Loss: 0.013421464711427689
Epoch: 6317 Loss: 0.013421151787042618
Epoch: 6318 Loss: 0.013416063971817493
Epoch: 6319 Loss: 0.013414276763796806
Epoch: 6320 Loss: 0.013409057632088661
Epoch: 6321 Loss: 0.0134047232568264
Epoch: 6322 Loss: 0.013402100652456284
Epoch: 6323 Loss: 0.0133997593075037
Epoch: 6324 Loss: 0.013397708535194397
Epoch: 6325 Loss: 0.0133922528475523
Epoch: 6326 Loss: 0.013389956206083298
Epoch: 6327 Loss: 0.013386305421590805
Epoch: 6328 Loss: 0.01338307373225689
Epoch: 6329 Loss: 0.013381927274167538
Epoch: 6330 Loss: 0.013376527465879917
Epoch: 6331 Loss: 0.013370051980018616
Epoch: 6332 Loss: 0.013368839398026466
Epoch: 6333 Loss: 0.013364576734602451
Epoch: 6334 Loss: 0.013362779282033443
Epoch: 6335 Loss: 0.013361683115363121
Epoch: 6336 Loss: 0.013354687951505184
Epoch: 6337 Loss: 0.013353751972317696
Epoch: 6338 Loss: 0.013347877189517021
Epoch: 6339 Loss: 0.013346857391297817
Epoch: 6340 Loss: 0.013344724662601948
Epoch: 6341 Loss: 0.0133402980864048
Epoch: 6342 Loss: 0.01333519909530878
Epoch: 6343 Loss: 0.013331837020814419
Epoch: 6344 Loss: 0.013330277055501938
Epoch: 6345 Loss: 0.01332901231944561
Epoch: 6346 Loss: 0.01332115475088358
Epoch: 6347 Loss: 0.013319524005055428
Epoch: 6348 Loss: 0.013315101154148579
Epoch: 6349 Loss: 0.013313843868672848
Epoch: 6350 Loss: 0.013309808447957039
Epoch: 6351 Loss: 0.013308876194059849
Epoch: 6352 Loss: 0.013302739709615707
Epoch: 6353 Loss: 0.013297965750098228
Epoch: 6354 Loss: 0.013295129872858524
Epoch: 6355 Loss: 0.013293311931192875
Epoch: 6356 Loss: 0.01329199131578207
Epoch: 6357 Loss: 0.01328529417514801
Epoch: 6358 Loss: 0.01328316517174244
Epoch: 6359 Loss: 0.013279112987220287
Epoch: 6360 Loss: 0.01327482145279646
Epoch: 6361 Loss: 0.013275455683469772
Epoch: 6362 Loss: 0.013270989060401917
Epoch: 6363 Loss: 0.013264498673379421
Epoch: 6364 Loss: 0.013260886073112488
Epoch: 6365 Loss: 0.013259612023830414
Epoch: 6366 Loss: 0.013255469501018524
Epoch: 6367 Loss: 0.013254527002573013
Epoch: 6368 Loss: 0.013248156756162643
Epoch: 6369 Loss: 0.013245231471955776
Epoch: 6370 Loss: 0.01324361003935337
Epoch: 6371 Loss: 0.013239027000963688
Epoch: 6372 Loss: 0.013236848637461662
Epoch: 6373 Loss: 0.01323562953621149
Epoch: 6374 Loss: 0.013229377567768097
Epoch: 6375 Loss: 0.013226788491010666
Epoch: 6376 Loss: 0.013221957720816135
Epoch: 6377 Loss: 0.013218485750257969
Epoch: 6378 Loss: 0.013216263614594936
Epoch: 6379 Loss: 0.013214931823313236
Epoch: 6380 Loss: 0.013210342265665531
Epoch: 6381 Loss: 0.013206999748945236
Epoch: 6382 Loss: 0.013204926624894142
Epoch: 6383 Loss: 0.01319887675344944
Epoch: 6384 Loss: 0.013197442516684532
Epoch: 6385 Loss: 0.013193580321967602
Epoch: 6386 Loss: 0.013191030360758305
Epoch: 6387 Loss: 0.01318776048719883
Epoch: 6388 Loss: 0.013184146955609322
Epoch: 6389 Loss: 0.01318241748958826
Epoch: 6390 Loss: 0.013176864013075829
Epoch: 6391 Loss: 0.013173086568713188
Epoch: 6392 Loss: 0.01317030843347311
Epoch: 6393 Loss: 0.01316883135586977
Epoch: 6394 Loss: 0.013166280463337898
Epoch: 6395 Loss: 0.0131624024361372
Epoch: 6396 Loss: 0.01315708365291357
Epoch: 6397 Loss: 0.013154142536222935
Epoch: 6398 Loss: 0.013149229809641838
Epoch: 6399 Loss: 0.013150031678378582
Epoch: 6400 Loss: 0.013143396936357021
Epoch: 6401 Loss: 0.013141157105565071
Epoch: 6402 Loss: 0.01313939318060875
Epoch: 6403 Loss: 0.013136927969753742
Epoch: 6404 Loss: 0.013130481354892254
Epoch: 6405 Loss: 0.013128808699548244
Epoch: 6406 Loss: 0.013124063611030579
Epoch: 6407 Loss: 0.013122967444360256
Epoch: 6408 Loss: 0.013118037953972816
Epoch: 6409 Loss: 0.013116099871695042
Epoch: 6410 Loss: 0.01311393454670906
Epoch: 6411 Loss: 0.013107595033943653
Epoch: 6412 Loss: 0.013105037622153759
Epoch: 6413 Loss: 0.013103769160807133
Epoch: 6414 Loss: 0.013100127689540386
Epoch: 6415 Loss: 0.01309932954609394
Epoch: 6416 Loss: 0.013093119487166405
Epoch: 6417 Loss: 0.01308963168412447
Epoch: 6418 Loss: 0.01308607216924429
Epoch: 6419 Loss: 0.01308511570096016
Epoch: 6420 Loss: 0.013080981560051441
Epoch: 6421 Loss: 0.013075930997729301
Epoch: 6422 Loss: 0.013072346337139606
Epoch: 6423 Loss: 0.013069993816316128
Epoch: 6424 Loss: 0.013066046871244907
Epoch: 6425 Loss: 0.013061793521046638
Epoch: 6426 Loss: 0.013060097582638264
Epoch: 6427 Loss: 0.013058477081358433
Epoch: 6428 Loss: 0.01305431593209505
Epoch: 6429 Loss: 0.013052103109657764
Epoch: 6430 Loss: 0.013050667010247707
Epoch: 6431 Loss: 0.0130465067923069
Epoch: 6432 Loss: 0.013042259030044079
Epoch: 6433 Loss: 0.013037054799497128
Epoch: 6434 Loss: 0.013035640120506287
Epoch: 6435 Loss: 0.013034453615546227
Epoch: 6436 Loss: 0.013030479662120342
Epoch: 6437 Loss: 0.013025355525314808
Epoch: 6438 Loss: 0.013022255152463913
Epoch: 6439 Loss: 0.013018667697906494
Epoch: 6440 Loss: 0.013016066513955593
Epoch: 6441 Loss: 0.013011815957725048
Epoch: 6442 Loss: 0.013010579161345959
Epoch: 6443 Loss: 0.013008363544940948
Epoch: 6444 Loss: 0.01300552487373352
Epoch: 6445 Loss: 0.013001018203794956
Epoch: 6446 Loss: 0.012996140867471695
Epoch: 6447 Loss: 0.012993178330361843
Epoch: 6448 Loss: 0.012991215102374554
Epoch: 6449 Loss: 0.012986578047275543
Epoch: 6450 Loss: 0.012985619716346264
Epoch: 6451 Loss: 0.012980850413441658
Epoch: 6452 Loss: 0.012977606616914272
Epoch: 6453 Loss: 0.012974489480257034
Epoch: 6454 Loss: 0.012974031269550323
Epoch: 6455 Loss: 0.012967870570719242
Epoch: 6456 Loss: 0.012965722009539604
Epoch: 6457 Loss: 0.012961440719664097
Epoch: 6458 Loss: 0.012959160842001438
Epoch: 6459 Loss: 0.01295478641986847
Epoch: 6460 Loss: 0.012953157536685467
Epoch: 6461 Loss: 0.012948513962328434
Epoch: 6462 Loss: 0.012946836650371552
Epoch: 6463 Loss: 0.012943915091454983
Epoch: 6464 Loss: 0.012942273169755936
Epoch: 6465 Loss: 0.012936258688569069
Epoch: 6466 Loss: 0.012932960875332355
Epoch: 6467 Loss: 0.012929359450936317
Epoch: 6468 Loss: 0.012927391566336155
Epoch: 6469 Loss: 0.01292529795318842
Epoch: 6470 Loss: 0.012921507470309734
Epoch: 6471 Loss: 0.012916726060211658
Epoch: 6472 Loss: 0.0129150515422225
Epoch: 6473 Loss: 0.012912525795400143
Epoch: 6474 Loss: 0.012908854521811008
Epoch: 6475 Loss: 0.012906298041343689
Epoch: 6476 Loss: 0.0129010658711195
Epoch: 6477 Loss: 0.012897356413304806
Epoch: 6478 Loss: 0.012896763160824776
Epoch: 6479 Loss: 0.012891007587313652
Epoch: 6480 Loss: 0.01289008092135191
Epoch: 6481 Loss: 0.012887369841337204
Epoch: 6482 Loss: 0.01288236491382122
Epoch: 6483 Loss: 0.01287874672561884
Epoch: 6484 Loss: 0.012877046130597591
Epoch: 6485 Loss: 0.012872789055109024
Epoch: 6486 Loss: 0.012869816273450851
Epoch: 6487 Loss: 0.012867157347500324
Epoch: 6488 Loss: 0.012865009717643261
Epoch: 6489 Loss: 0.012861452996730804
Epoch: 6490 Loss: 0.01285707950592041
Epoch: 6491 Loss: 0.0128554105758667
Epoch: 6492 Loss: 0.012854045256972313
Epoch: 6493 Loss: 0.012849299237132072
Epoch: 6494 Loss: 0.012845208868384361
Epoch: 6495 Loss: 0.012843973003327847
Epoch: 6496 Loss: 0.012840609066188335
Epoch: 6497 Loss: 0.012835720553994179
Epoch: 6498 Loss: 0.01283333357423544
Epoch: 6499 Loss: 0.012831716798245907
Epoch: 6500 Loss: 0.012826691381633282
Epoch: 6501 Loss: 0.01282449346035719
Epoch: 6502 Loss: 0.012822302058339119
Epoch: 6503 Loss: 0.012818915769457817
Epoch: 6504 Loss: 0.012815631926059723
Epoch: 6505 Loss: 0.012811319902539253
Epoch: 6506 Loss: 0.01280969101935625
Epoch: 6507 Loss: 0.012804276309907436
Epoch: 6508 Loss: 0.01280203741043806
Epoch: 6509 Loss: 0.012798270210623741
Epoch: 6510 Loss: 0.012794727459549904
Epoch: 6511 Loss: 0.012792662717401981
Epoch: 6512 Loss: 0.012789742089807987
Epoch: 6513 Loss: 0.012787813320755959
Epoch: 6514 Loss: 0.012783730402588844
Epoch: 6515 Loss: 0.012780445627868176
Epoch: 6516 Loss: 0.012776385992765427
Epoch: 6517 Loss: 0.012776065617799759
Epoch: 6518 Loss: 0.012772354297339916
Epoch: 6519 Loss: 0.012768305838108063
Epoch: 6520 Loss: 0.012764805927872658
Epoch: 6521 Loss: 0.01276226807385683
Epoch: 6522 Loss: 0.012758365832269192
Epoch: 6523 Loss: 0.01275621261447668
Epoch: 6524 Loss: 0.012754034250974655
Epoch: 6525 Loss: 0.012748281471431255
Epoch: 6526 Loss: 0.01274662371724844
Epoch: 6527 Loss: 0.012744489125907421
Epoch: 6528 Loss: 0.012741167098283768
Epoch: 6529 Loss: 0.012739220634102821
Epoch: 6530 Loss: 0.012736191973090172
Epoch: 6531 Loss: 0.012730127200484276
Epoch: 6532 Loss: 0.012726877816021442
Epoch: 6533 Loss: 0.012724272906780243
Epoch: 6534 Loss: 0.01272312831133604
Epoch: 6535 Loss: 0.012719349004328251
Epoch: 6536 Loss: 0.012714977376163006
Epoch: 6537 Loss: 0.012712348252534866
Epoch: 6538 Loss: 0.012708807364106178
Epoch: 6539 Loss: 0.012706012465059757
Epoch: 6540 Loss: 0.012704559601843357
Epoch: 6541 Loss: 0.012702496722340584
Epoch: 6542 Loss: 0.012697975151240826
Epoch: 6543 Loss: 0.012694934383034706
Epoch: 6544 Loss: 0.012690898962318897
Epoch: 6545 Loss: 0.012687807902693748
Epoch: 6546 Loss: 0.01268615573644638
Epoch: 6547 Loss: 0.012683968059718609
Epoch: 6548 Loss: 0.012680702842772007
Epoch: 6549 Loss: 0.012675694189965725
Epoch: 6550 Loss: 0.01267272513359785
Epoch: 6551 Loss: 0.012670769356191158
Epoch: 6552 Loss: 0.012666597031056881
Epoch: 6553 Loss: 0.012665635906159878
Epoch: 6554 Loss: 0.012660976499319077
Epoch: 6555 Loss: 0.012657899409532547
Epoch: 6556 Loss: 0.012658610939979553
Epoch: 6557 Loss: 0.012652597390115261
Epoch: 6558 Loss: 0.01265024021267891
Epoch: 6559 Loss: 0.012645719572901726
Epoch: 6560 Loss: 0.012643404304981232
Epoch: 6561 Loss: 0.012639453634619713
Epoch: 6562 Loss: 0.012638538144528866
Epoch: 6563 Loss: 0.012634974904358387
Epoch: 6564 Loss: 0.012630260549485683
Epoch: 6565 Loss: 0.012628086842596531
Epoch: 6566 Loss: 0.012624435126781464
Epoch: 6567 Loss: 0.012621117755770683
Epoch: 6568 Loss: 0.01261770911514759
Epoch: 6569 Loss: 0.012614496052265167
Epoch: 6570 Loss: 0.012612276710569859
Epoch: 6571 Loss: 0.01261085830628872
Epoch: 6572 Loss: 0.012605850584805012
Epoch: 6573 Loss: 0.012604555115103722
Epoch: 6574 Loss: 0.012601708061993122
Epoch: 6575 Loss: 0.01259952038526535
Epoch: 6576 Loss: 0.012595051899552345
Epoch: 6577 Loss: 0.012590751051902771
Epoch: 6578 Loss: 0.012588951736688614
Epoch: 6579 Loss: 0.01258605346083641
Epoch: 6580 Loss: 0.012583175674080849
Epoch: 6581 Loss: 0.01258083526045084
Epoch: 6582 Loss: 0.012578237801790237
Epoch: 6583 Loss: 0.012575099244713783
Epoch: 6584 Loss: 0.012570381164550781
Epoch: 6585 Loss: 0.01256731990724802
Epoch: 6586 Loss: 0.01256562490016222
Epoch: 6587 Loss: 0.012562859803438187
Epoch: 6588 Loss: 0.012559127993881702
Epoch: 6589 Loss: 0.012555267661809921
Epoch: 6590 Loss: 0.012553022243082523
Epoch: 6591 Loss: 0.01255008578300476
Epoch: 6592 Loss: 0.01254628598690033
Epoch: 6593 Loss: 0.012545708566904068
Epoch: 6594 Loss: 0.012541593983769417
Epoch: 6595 Loss: 0.012536871246993542
Epoch: 6596 Loss: 0.0125348549336195
Epoch: 6597 Loss: 0.01253058947622776
Epoch: 6598 Loss: 0.012528972700238228
Epoch: 6599 Loss: 0.012528485618531704
Epoch: 6600 Loss: 0.012525462545454502
Epoch: 6601 Loss: 0.012519163079559803
Epoch: 6602 Loss: 0.01251666434109211
Epoch: 6603 Loss: 0.012513989582657814
Epoch: 6604 Loss: 0.012510288506746292
Epoch: 6605 Loss: 0.012509006075561047
Epoch: 6606 Loss: 0.012504134327173233
Epoch: 6607 Loss: 0.012502058409154415
Epoch: 6608 Loss: 0.012498264200985432
Epoch: 6609 Loss: 0.012496045790612698
Epoch: 6610 Loss: 0.012494124472141266
Epoch: 6611 Loss: 0.012492172420024872
Epoch: 6612 Loss: 0.012487173080444336
Epoch: 6613 Loss: 0.01248506736010313
Epoch: 6614 Loss: 0.01248253509402275
Epoch: 6615 Loss: 0.01247906032949686
Epoch: 6616 Loss: 0.01247588824480772
Epoch: 6617 Loss: 0.012473021633923054
Epoch: 6618 Loss: 0.012470097281038761
Epoch: 6619 Loss: 0.012466493062675
Epoch: 6620 Loss: 0.012461845763027668
Epoch: 6621 Loss: 0.012460852973163128
Epoch: 6622 Loss: 0.012457909993827343
Epoch: 6623 Loss: 0.012456252239644527
Epoch: 6624 Loss: 0.012451902031898499
Epoch: 6625 Loss: 0.012450100854039192
Epoch: 6626 Loss: 0.012445534579455853
Epoch: 6627 Loss: 0.012441836297512054
Epoch: 6628 Loss: 0.012439745478332043
Epoch: 6629 Loss: 0.012437294237315655
Epoch: 6630 Loss: 0.012434429489076138
Epoch: 6631 Loss: 0.012433027848601341
Epoch: 6632 Loss: 0.012429307214915752
Epoch: 6633 Loss: 0.012425152584910393
Epoch: 6634 Loss: 0.012421783991158009
Epoch: 6635 Loss: 0.012420352548360825
Epoch: 6636 Loss: 0.012417830526828766
Epoch: 6637 Loss: 0.012413663789629936
Epoch: 6638 Loss: 0.012410713359713554
Epoch: 6639 Loss: 0.012406737543642521
Epoch: 6640 Loss: 0.012403653934597969
Epoch: 6641 Loss: 0.012402198277413845
Epoch: 6642 Loss: 0.01239839754998684
Epoch: 6643 Loss: 0.012396714650094509
Epoch: 6644 Loss: 0.012394430115818977
Epoch: 6645 Loss: 0.012388578616082668
Epoch: 6646 Loss: 0.012386532500386238
Epoch: 6647 Loss: 0.012383242137730122
Epoch: 6648 Loss: 0.012380402535200119
Epoch: 6649 Loss: 0.012377768754959106
Epoch: 6650 Loss: 0.012377426959574223
Epoch: 6651 Loss: 0.01237235963344574
Epoch: 6652 Loss: 0.012371963821351528
Epoch: 6653 Loss: 0.012366174720227718
Epoch: 6654 Loss: 0.012362580746412277
Epoch: 6655 Loss: 0.012360930442810059
Epoch: 6656 Loss: 0.012359538115561008
Epoch: 6657 Loss: 0.012357614003121853
Epoch: 6658 Loss: 0.012356027029454708
Epoch: 6659 Loss: 0.012348806485533714
Epoch: 6660 Loss: 0.012346658855676651
Epoch: 6661 Loss: 0.0123427864164114
Epoch: 6662 Loss: 0.012343153357505798
Epoch: 6663 Loss: 0.012335984967648983
Epoch: 6664 Loss: 0.012334632687270641
Epoch: 6665 Loss: 0.012330519035458565
Epoch: 6666 Loss: 0.012327865697443485
Epoch: 6667 Loss: 0.01232414785772562
Epoch: 6668 Loss: 0.012325506657361984
Epoch: 6669 Loss: 0.01232108473777771
Epoch: 6670 Loss: 0.012317916378378868
Epoch: 6671 Loss: 0.012314044870436192
Epoch: 6672 Loss: 0.012312899343669415
Epoch: 6673 Loss: 0.012307470664381981
Epoch: 6674 Loss: 0.012304017320275307
Epoch: 6675 Loss: 0.012302303686738014
Epoch: 6676 Loss: 0.01229860819876194
Epoch: 6677 Loss: 0.012295820750296116
Epoch: 6678 Loss: 0.012295495718717575
Epoch: 6679 Loss: 0.012292871251702309
Epoch: 6680 Loss: 0.012290816754102707
Epoch: 6681 Loss: 0.012283607386052608
Epoch: 6682 Loss: 0.012281843461096287
Epoch: 6683 Loss: 0.012280509807169437
Epoch: 6684 Loss: 0.012276401743292809
Epoch: 6685 Loss: 0.012274102307856083
Epoch: 6686 Loss: 0.012270348146557808
Epoch: 6687 Loss: 0.012267866171896458
Epoch: 6688 Loss: 0.012265220284461975
Epoch: 6689 Loss: 0.012261854484677315
Epoch: 6690 Loss: 0.012260783463716507
Epoch: 6691 Loss: 0.01225784607231617
Epoch: 6692 Loss: 0.012254360131919384
Epoch: 6693 Loss: 0.012250429019331932
Epoch: 6694 Loss: 0.012248405255377293
Epoch: 6695 Loss: 0.012245358899235725
Epoch: 6696 Loss: 0.012244150973856449
Epoch: 6697 Loss: 0.012239562347531319
Epoch: 6698 Loss: 0.012236564420163631
Epoch: 6699 Loss: 0.012234577909111977
Epoch: 6700 Loss: 0.012231266126036644
Epoch: 6701 Loss: 0.012228571809828281
Epoch: 6702 Loss: 0.012223326601088047
Epoch: 6703 Loss: 0.01222264114767313
Epoch: 6704 Loss: 0.012217788957059383
Epoch: 6705 Loss: 0.012217951938509941
Epoch: 6706 Loss: 0.012212520465254784
Epoch: 6707 Loss: 0.012210840359330177
Epoch: 6708 Loss: 0.012208113446831703
Epoch: 6709 Loss: 0.012205302715301514
Epoch: 6710 Loss: 0.012200569733977318
Epoch: 6711 Loss: 0.012200022116303444
Epoch: 6712 Loss: 0.012196515686810017
Epoch: 6713 Loss: 0.01219471450895071
Epoch: 6714 Loss: 0.01219142135232687
Epoch: 6715 Loss: 0.012189796194434166
Epoch: 6716 Loss: 0.01218388881534338
Epoch: 6717 Loss: 0.012182043865323067
Epoch: 6718 Loss: 0.012179373763501644
Epoch: 6719 Loss: 0.012176022864878178
Epoch: 6720 Loss: 0.01217318419367075
Epoch: 6721 Loss: 0.012173582799732685
Epoch: 6722 Loss: 0.012169129215180874
Epoch: 6723 Loss: 0.012164266780018806
Epoch: 6724 Loss: 0.012161238119006157
Epoch: 6725 Loss: 0.012159847654402256
Epoch: 6726 Loss: 0.012157825753092766
Epoch: 6727 Loss: 0.012153923511505127
Epoch: 6728 Loss: 0.012151825241744518
Epoch: 6729 Loss: 0.012150428257882595
Epoch: 6730 Loss: 0.01214484591037035
Epoch: 6731 Loss: 0.012143473140895367
Epoch: 6732 Loss: 0.012140053324401379
Epoch: 6733 Loss: 0.012137114070355892
Epoch: 6734 Loss: 0.012137227691709995
Epoch: 6735 Loss: 0.012133372016251087
Epoch: 6736 Loss: 0.012129480019211769
Epoch: 6737 Loss: 0.012125417590141296
Epoch: 6738 Loss: 0.01212270837277174
Epoch: 6739 Loss: 0.012120532803237438
Epoch: 6740 Loss: 0.01211873721331358
Epoch: 6741 Loss: 0.012114702723920345
Epoch: 6742 Loss: 0.01211195345968008
Epoch: 6743 Loss: 0.012108312919735909
Epoch: 6744 Loss: 0.012106121517717838
Epoch: 6745 Loss: 0.01210414431989193
Epoch: 6746 Loss: 0.012101435102522373
Epoch: 6747 Loss: 0.01209886372089386
Epoch: 6748 Loss: 0.012094038538634777
Epoch: 6749 Loss: 0.012092283926904202
Epoch: 6750 Loss: 0.012088482268154621
Epoch: 6751 Loss: 0.012084498070180416
Epoch: 6752 Loss: 0.012085525318980217
Epoch: 6753 Loss: 0.012081572785973549
Epoch: 6754 Loss: 0.01207886915653944
Epoch: 6755 Loss: 0.01207619160413742
Epoch: 6756 Loss: 0.012071714736521244
Epoch: 6757 Loss: 0.012068232521414757
Epoch: 6758 Loss: 0.012067890726029873
Epoch: 6759 Loss: 0.012065312825143337
Epoch: 6760 Loss: 0.012062192894518375
Epoch: 6761 Loss: 0.0120598915964365
Epoch: 6762 Loss: 0.01205519214272499
Epoch: 6763 Loss: 0.012053001672029495
Epoch: 6764 Loss: 0.012049231678247452
Epoch: 6765 Loss: 0.01204789150506258
Epoch: 6766 Loss: 0.012043043039739132
Epoch: 6767 Loss: 0.0120437266305089
Epoch: 6768 Loss: 0.012038767337799072
Epoch: 6769 Loss: 0.012036307714879513
Epoch: 6770 Loss: 0.012033740058541298
Epoch: 6771 Loss: 0.012030593119561672
Epoch: 6772 Loss: 0.012027123011648655
Epoch: 6773 Loss: 0.012026106007397175
Epoch: 6774 Loss: 0.012023761868476868
Epoch: 6775 Loss: 0.012020974420011044
Epoch: 6776 Loss: 0.012018507346510887
Epoch: 6777 Loss: 0.01201536227017641
Epoch: 6778 Loss: 0.012010671198368073
Epoch: 6779 Loss: 0.012007889337837696
Epoch: 6780 Loss: 0.012006226927042007
Epoch: 6781 Loss: 0.012003077194094658
Epoch: 6782 Loss: 0.012002041563391685
Epoch: 6783 Loss: 0.011998225934803486
Epoch: 6784 Loss: 0.011994360946118832
Epoch: 6785 Loss: 0.011991467326879501
Epoch: 6786 Loss: 0.01198799442499876
Epoch: 6787 Loss: 0.011984617449343204
Epoch: 6788 Loss: 0.011984400451183319
Epoch: 6789 Loss: 0.011980515904724598
Epoch: 6790 Loss: 0.011976968497037888
Epoch: 6791 Loss: 0.011974290013313293
Epoch: 6792 Loss: 0.011972924694418907
Epoch: 6793 Loss: 0.011969562619924545
Epoch: 6794 Loss: 0.011969316750764847
Epoch: 6795 Loss: 0.01196403056383133
Epoch: 6796 Loss: 0.011963135562837124
Epoch: 6797 Loss: 0.011958318762481213
Epoch: 6798 Loss: 0.011956118047237396
Epoch: 6799 Loss: 0.011952894739806652
Epoch: 6800 Loss: 0.011949820443987846
Epoch: 6801 Loss: 0.011946308426558971
Epoch: 6802 Loss: 0.011946459300816059
Epoch: 6803 Loss: 0.011943744495511055
Epoch: 6804 Loss: 0.011938587762415409
Epoch: 6805 Loss: 0.011936882510781288
Epoch: 6806 Loss: 0.011935940012335777
Epoch: 6807 Loss: 0.01193139236420393
Epoch: 6808 Loss: 0.011929244734346867
Epoch: 6809 Loss: 0.01192657183855772
Epoch: 6810 Loss: 0.011923661455512047
Epoch: 6811 Loss: 0.011919036507606506
Epoch: 6812 Loss: 0.011916466988623142
Epoch: 6813 Loss: 0.011912493966519833
Epoch: 6814 Loss: 0.011911559849977493
Epoch: 6815 Loss: 0.01190910767763853
Epoch: 6816 Loss: 0.011907080188393593
Epoch: 6817 Loss: 0.011905280873179436
Epoch: 6818 Loss: 0.01190353836864233
Epoch: 6819 Loss: 0.011898509226739407
Epoch: 6820 Loss: 0.011894524097442627
Epoch: 6821 Loss: 0.011893345974385738
Epoch: 6822 Loss: 0.011889530345797539
Epoch: 6823 Loss: 0.011888553388416767
Epoch: 6824 Loss: 0.011886414140462875
Epoch: 6825 Loss: 0.01188195962458849
Epoch: 6826 Loss: 0.011878091841936111
Epoch: 6827 Loss: 0.011877698823809624
Epoch: 6828 Loss: 0.011873230338096619
Epoch: 6829 Loss: 0.011870245449244976
Epoch: 6830 Loss: 0.011869462206959724
Epoch: 6831 Loss: 0.011865838430821896
Epoch: 6832 Loss: 0.01186196506023407
Epoch: 6833 Loss: 0.011859800666570663
Epoch: 6834 Loss: 0.011859877966344357
Epoch: 6835 Loss: 0.011854968965053558
Epoch: 6836 Loss: 0.011854449287056923
Epoch: 6837 Loss: 0.011851075105369091
Epoch: 6838 Loss: 0.01184498518705368
Epoch: 6839 Loss: 0.011844949796795845
Epoch: 6840 Loss: 0.01184147596359253
Epoch: 6841 Loss: 0.011838854290544987
Epoch: 6842 Loss: 0.01183631457388401
Epoch: 6843 Loss: 0.011834485456347466
Epoch: 6844 Loss: 0.011830641888082027
Epoch: 6845 Loss: 0.011829501949250698
Epoch: 6846 Loss: 0.011824890971183777
Epoch: 6847 Loss: 0.01182234100997448
Epoch: 6848 Loss: 0.011818613857030869
Epoch: 6849 Loss: 0.011817904189229012
Epoch: 6850 Loss: 0.011817166581749916
Epoch: 6851 Loss: 0.01181221567094326
Epoch: 6852 Loss: 0.011809087358415127
Epoch: 6853 Loss: 0.011806385591626167
Epoch: 6854 Loss: 0.011805898509919643
Epoch: 6855 Loss: 0.011801538988947868
Epoch: 6856 Loss: 0.01179732009768486
Epoch: 6857 Loss: 0.01179688423871994
Epoch: 6858 Loss: 0.011795268394052982
Epoch: 6859 Loss: 0.011791159398853779
Epoch: 6860 Loss: 0.011787932366132736
Epoch: 6861 Loss: 0.011785067617893219
Epoch: 6862 Loss: 0.011781381443142891
Epoch: 6863 Loss: 0.011779345571994781
Epoch: 6864 Loss: 0.0117778480052948
Epoch: 6865 Loss: 0.011773851700127125
Epoch: 6866 Loss: 0.01177096925675869
Epoch: 6867 Loss: 0.011769481934607029
Epoch: 6868 Loss: 0.011766484938561916
Epoch: 6869 Loss: 0.01176338642835617
Epoch: 6870 Loss: 0.011762995272874832
Epoch: 6871 Loss: 0.011760170571506023
Epoch: 6872 Loss: 0.011755392886698246
Epoch: 6873 Loss: 0.01175379753112793
Epoch: 6874 Loss: 0.011749031953513622
Epoch: 6875 Loss: 0.011749944649636745
Epoch: 6876 Loss: 0.011745048686861992
Epoch: 6877 Loss: 0.011743451468646526
Epoch: 6878 Loss: 0.011740006506443024
Epoch: 6879 Loss: 0.011737709864974022
Epoch: 6880 Loss: 0.011733238585293293
Epoch: 6881 Loss: 0.01173142995685339
Epoch: 6882 Loss: 0.011727830395102501
Epoch: 6883 Loss: 0.01172570139169693
Epoch: 6884 Loss: 0.011724785901606083
Epoch: 6885 Loss: 0.011722208932042122
Epoch: 6886 Loss: 0.011718233115971088
Epoch: 6887 Loss: 0.011715009808540344
Epoch: 6888 Loss: 0.011713332496583462
Epoch: 6889 Loss: 0.011711054481565952
Epoch: 6890 Loss: 0.011708885431289673
Epoch: 6891 Loss: 0.011705621145665646
Epoch: 6892 Loss: 0.0117022804915905
Epoch: 6893 Loss: 0.011699008755385876
Epoch: 6894 Loss: 0.011696643196046352
Epoch: 6895 Loss: 0.011695572175085545
Epoch: 6896 Loss: 0.011692261323332787
Epoch: 6897 Loss: 0.011688527651131153
Epoch: 6898 Loss: 0.01168585941195488
Epoch: 6899 Loss: 0.011683943681418896
Epoch: 6900 Loss: 0.011681301519274712
Epoch: 6901 Loss: 0.011679714545607567
Epoch: 6902 Loss: 0.011677086353302002
Epoch: 6903 Loss: 0.011674381792545319
Epoch: 6904 Loss: 0.011669712141156197
Epoch: 6905 Loss: 0.011668517254292965
Epoch: 6906 Loss: 0.01166495867073536
Epoch: 6907 Loss: 0.011663108132779598
Epoch: 6908 Loss: 0.011661055497825146
Epoch: 6909 Loss: 0.011657387018203735
Epoch: 6910 Loss: 0.011654797941446304
Epoch: 6911 Loss: 0.011652604676783085
Epoch: 6912 Loss: 0.011650425381958485
Epoch: 6913 Loss: 0.011649451218545437
Epoch: 6914 Loss: 0.011646689847111702
Epoch: 6915 Loss: 0.011642899364233017
Epoch: 6916 Loss: 0.011638587340712547
Epoch: 6917 Loss: 0.011638335883617401
Epoch: 6918 Loss: 0.011633862741291523
Epoch: 6919 Loss: 0.01163287553936243
Epoch: 6920 Loss: 0.011629297398030758
Epoch: 6921 Loss: 0.011627170257270336
Epoch: 6922 Loss: 0.011624193750321865
Epoch: 6923 Loss: 0.011621417477726936
Epoch: 6924 Loss: 0.01161891594529152
Epoch: 6925 Loss: 0.011616086587309837
Epoch: 6926 Loss: 0.011614575982093811
Epoch: 6927 Loss: 0.011609546840190887
Epoch: 6928 Loss: 0.011608118191361427
Epoch: 6929 Loss: 0.011605723761022091
Epoch: 6930 Loss: 0.011602433398365974
Epoch: 6931 Loss: 0.01160095538944006
Epoch: 6932 Loss: 0.011597087606787682
Epoch: 6933 Loss: 0.011594610288739204
Epoch: 6934 Loss: 0.011593420058488846
Epoch: 6935 Loss: 0.011588982306420803
Epoch: 6936 Loss: 0.011587303131818771
Epoch: 6937 Loss: 0.011584209278225899
Epoch: 6938 Loss: 0.011580444872379303
Epoch: 6939 Loss: 0.011579258367419243
Epoch: 6940 Loss: 0.011576378718018532
Epoch: 6941 Loss: 0.011573954485356808
Epoch: 6942 Loss: 0.011570616625249386
Epoch: 6943 Loss: 0.011571207083761692
Epoch: 6944 Loss: 0.011565588414669037
Epoch: 6945 Loss: 0.011563794687390327
Epoch: 6946 Loss: 0.011560535989701748
Epoch: 6947 Loss: 0.011557185091078281
Epoch: 6948 Loss: 0.011556502431631088
Epoch: 6949 Loss: 0.011552682146430016
Epoch: 6950 Loss: 0.011552279815077782
Epoch: 6951 Loss: 0.011547735892236233
Epoch: 6952 Loss: 0.011545142158865929
Epoch: 6953 Loss: 0.011543234810233116
Epoch: 6954 Loss: 0.011542870663106441
Epoch: 6955 Loss: 0.01153904665261507
Epoch: 6956 Loss: 0.011534749530255795
Epoch: 6957 Loss: 0.011531510390341282
Epoch: 6958 Loss: 0.0115306805819273
Epoch: 6959 Loss: 0.011527402326464653
Epoch: 6960 Loss: 0.01152439322322607
Epoch: 6961 Loss: 0.011522043496370316
Epoch: 6962 Loss: 0.011518572457134724
Epoch: 6963 Loss: 0.011518646962940693
Epoch: 6964 Loss: 0.01151473168283701
Epoch: 6965 Loss: 0.011512551456689835
Epoch: 6966 Loss: 0.011509856209158897
Epoch: 6967 Loss: 0.011507408693432808
Epoch: 6968 Loss: 0.011504828929901123
Epoch: 6969 Loss: 0.011502600274980068
Epoch: 6970 Loss: 0.01150041539222002
Epoch: 6971 Loss: 0.011496658436954021
Epoch: 6972 Loss: 0.011493782512843609
Epoch: 6973 Loss: 0.011490777134895325
Epoch: 6974 Loss: 0.011488995514810085
Epoch: 6975 Loss: 0.01148867979645729
Epoch: 6976 Loss: 0.011484003625810146
Epoch: 6977 Loss: 0.01148250326514244
Epoch: 6978 Loss: 0.011478058062493801
Epoch: 6979 Loss: 0.011476010084152222
Epoch: 6980 Loss: 0.011473790742456913
Epoch: 6981 Loss: 0.011471806094050407
Epoch: 6982 Loss: 0.011468971148133278
Epoch: 6983 Loss: 0.01146719977259636
Epoch: 6984 Loss: 0.011462517082691193
Epoch: 6985 Loss: 0.01146051287651062
Epoch: 6986 Loss: 0.01146014779806137
Epoch: 6987 Loss: 0.011456362903118134
Epoch: 6988 Loss: 0.011452682316303253
Epoch: 6989 Loss: 0.011451604776084423
Epoch: 6990 Loss: 0.011447924189269543
Epoch: 6991 Loss: 0.01144526619464159
Epoch: 6992 Loss: 0.011442952789366245
Epoch: 6993 Loss: 0.011440248228609562
Epoch: 6994 Loss: 0.011439359746873379
Epoch: 6995 Loss: 0.011435535736382008
Epoch: 6996 Loss: 0.01143502164632082
Epoch: 6997 Loss: 0.011432120576500893
Epoch: 6998 Loss: 0.011426921933889389
Epoch: 6999 Loss: 0.01142461970448494
Epoch: 7000 Loss: 0.011422261595726013
Epoch: 7001 Loss: 0.011420035734772682
Epoch: 7002 Loss: 0.011417240835726261
Epoch: 7003 Loss: 0.011416163295507431
Epoch: 7004 Loss: 0.01141445804387331
Epoch: 7005 Loss: 0.011411799117922783
Epoch: 7006 Loss: 0.01140686497092247
Epoch: 7007 Loss: 0.011405340395867825
Epoch: 7008 Loss: 0.01140343863517046
Epoch: 7009 Loss: 0.011400709860026836
Epoch: 7010 Loss: 0.011398951523005962
Epoch: 7011 Loss: 0.011394859291613102
Epoch: 7012 Loss: 0.011392547748982906
Epoch: 7013 Loss: 0.01139134168624878
Epoch: 7014 Loss: 0.011386587284505367
Epoch: 7015 Loss: 0.011385580524802208
Epoch: 7016 Loss: 0.01138269156217575
Epoch: 7017 Loss: 0.011379607953131199
Epoch: 7018 Loss: 0.011377325281500816
Epoch: 7019 Loss: 0.011375124566257
Epoch: 7020 Loss: 0.011374669149518013
Epoch: 7021 Loss: 0.011370605789124966
Epoch: 7022 Loss: 0.011367018334567547
Epoch: 7023 Loss: 0.011365520767867565
Epoch: 7024 Loss: 0.011362406425178051
Epoch: 7025 Loss: 0.011360316537320614
Epoch: 7026 Loss: 0.0113593814894557
Epoch: 7027 Loss: 0.01135660894215107
Epoch: 7028 Loss: 0.011351896449923515
Epoch: 7029 Loss: 0.011349738575518131
Epoch: 7030 Loss: 0.011347698979079723
Epoch: 7031 Loss: 0.011344240047037601
Epoch: 7032 Loss: 0.011344247497618198
Epoch: 7033 Loss: 0.011339952237904072
Epoch: 7034 Loss: 0.011336533352732658
Epoch: 7035 Loss: 0.01133561972528696
Epoch: 7036 Loss: 0.011332985013723373
Epoch: 7037 Loss: 0.011330245062708855
Epoch: 7038 Loss: 0.011326475068926811
Epoch: 7039 Loss: 0.0113241421058774
Epoch: 7040 Loss: 0.011321822181344032
Epoch: 7041 Loss: 0.011319832876324654
Epoch: 7042 Loss: 0.011317758820950985
Epoch: 7043 Loss: 0.01131486613303423
Epoch: 7044 Loss: 0.011313742026686668
Epoch: 7045 Loss: 0.0113097308203578
Epoch: 7046 Loss: 0.01130695827305317
Epoch: 7047 Loss: 0.011303647421300411
Epoch: 7048 Loss: 0.011302033439278603
Epoch: 7049 Loss: 0.011299795471131802
Epoch: 7050 Loss: 0.011297897435724735
Epoch: 7051 Loss: 0.011296006850898266
Epoch: 7052 Loss: 0.011292832903563976
Epoch: 7053 Loss: 0.011291181668639183
Epoch: 7054 Loss: 0.011287139728665352
Epoch: 7055 Loss: 0.011285428889095783
Epoch: 7056 Loss: 0.011283902451395988
Epoch: 7057 Loss: 0.011279821395874023
Epoch: 7058 Loss: 0.011277827434241772
Epoch: 7059 Loss: 0.011273883283138275
Epoch: 7060 Loss: 0.011273239739239216
Epoch: 7061 Loss: 0.011271476745605469
Epoch: 7062 Loss: 0.011267179623246193
Epoch: 7063 Loss: 0.011266774497926235
Epoch: 7064 Loss: 0.011262495070695877
Epoch: 7065 Loss: 0.011260217987000942
Epoch: 7066 Loss: 0.011257738806307316
Epoch: 7067 Loss: 0.011255526915192604
Epoch: 7068 Loss: 0.01125471293926239
Epoch: 7069 Loss: 0.01125013642013073
Epoch: 7070 Loss: 0.011246654205024242
Epoch: 7071 Loss: 0.01124515850096941
Epoch: 7072 Loss: 0.011242039501667023
Epoch: 7073 Loss: 0.011241321451961994
Epoch: 7074 Loss: 0.011238080449402332
Epoch: 7075 Loss: 0.011235238052904606
Epoch: 7076 Loss: 0.011234060861170292
Epoch: 7077 Loss: 0.011232463642954826
Epoch: 7078 Loss: 0.011227649636566639
Epoch: 7079 Loss: 0.011226178146898746
Epoch: 7080 Loss: 0.011222459375858307
Epoch: 7081 Loss: 0.011221353895962238
Epoch: 7082 Loss: 0.011221212334930897
Epoch: 7083 Loss: 0.01121564768254757
Epoch: 7084 Loss: 0.01121386606246233
Epoch: 7085 Loss: 0.011211653240025043
Epoch: 7086 Loss: 0.011208749376237392
Epoch: 7087 Loss: 0.011206835508346558
Epoch: 7088 Loss: 0.011204782873392105
Epoch: 7089 Loss: 0.011200757697224617
Epoch: 7090 Loss: 0.011198682710528374
Epoch: 7091 Loss: 0.011195991188287735
Epoch: 7092 Loss: 0.011193317361176014
Epoch: 7093 Loss: 0.011191727593541145
Epoch: 7094 Loss: 0.011188462376594543
Epoch: 7095 Loss: 0.01118653453886509
Epoch: 7096 Loss: 0.011187559925019741
Epoch: 7097 Loss: 0.011181412264704704
Epoch: 7098 Loss: 0.011179075576364994
Epoch: 7099 Loss: 0.011177204549312592
Epoch: 7100 Loss: 0.011175757274031639
Epoch: 7101 Loss: 0.01117231510579586
Epoch: 7102 Loss: 0.011170780286192894
Epoch: 7103 Loss: 0.011167055927217007
Epoch: 7104 Loss: 0.011165335774421692
Epoch: 7105 Loss: 0.011162015609443188
Epoch: 7106 Loss: 0.011160382069647312
Epoch: 7107 Loss: 0.011157892644405365
Epoch: 7108 Loss: 0.011154159903526306
Epoch: 7109 Loss: 0.011154203675687313
Epoch: 7110 Loss: 0.011149594560265541
Epoch: 7111 Loss: 0.01114821806550026
Epoch: 7112 Loss: 0.01114722527563572
Epoch: 7113 Loss: 0.011143720708787441
Epoch: 7114 Loss: 0.011140468530356884
Epoch: 7115 Loss: 0.01113822776824236
Epoch: 7116 Loss: 0.011136279441416264
Epoch: 7117 Loss: 0.011134687811136246
Epoch: 7118 Loss: 0.011131267994642258
Epoch: 7119 Loss: 0.011127883568406105
Epoch: 7120 Loss: 0.011125660501420498
Epoch: 7121 Loss: 0.011122521944344044
Epoch: 7122 Loss: 0.011122532188892365
Epoch: 7123 Loss: 0.01111928466707468
Epoch: 7124 Loss: 0.011116639710962772
Epoch: 7125 Loss: 0.011113506741821766
Epoch: 7126 Loss: 0.011112071573734283
Epoch: 7127 Loss: 0.0111085819080472
Epoch: 7128 Loss: 0.011106878519058228
Epoch: 7129 Loss: 0.011104613542556763
Epoch: 7130 Loss: 0.011102014221251011
Epoch: 7131 Loss: 0.011100439354777336
Epoch: 7132 Loss: 0.01109692919999361
Epoch: 7133 Loss: 0.01109444908797741
Epoch: 7134 Loss: 0.011092121712863445
Epoch: 7135 Loss: 0.01109143067151308
Epoch: 7136 Loss: 0.011088218539953232
Epoch: 7137 Loss: 0.011084255762398243
Epoch: 7138 Loss: 0.011082015000283718
Epoch: 7139 Loss: 0.011081743985414505
Epoch: 7140 Loss: 0.011077366769313812
Epoch: 7141 Loss: 0.011076955124735832
Epoch: 7142 Loss: 0.011073119007050991
Epoch: 7143 Loss: 0.011070415377616882
Epoch: 7144 Loss: 0.011067841202020645
Epoch: 7145 Loss: 0.01106681115925312
Epoch: 7146 Loss: 0.01106348354369402
Epoch: 7147 Loss: 0.01106309611350298
Epoch: 7148 Loss: 0.011058217845857143
Epoch: 7149 Loss: 0.011055666953325272
Epoch: 7150 Loss: 0.011053201742470264
Epoch: 7151 Loss: 0.0110520850867033
Epoch: 7152 Loss: 0.011049696244299412
Epoch: 7153 Loss: 0.011047173291444778
Epoch: 7154 Loss: 0.011047422885894775
Epoch: 7155 Loss: 0.011043223552405834
Epoch: 7156 Loss: 0.011039961129426956
Epoch: 7157 Loss: 0.01103696133941412
Epoch: 7158 Loss: 0.011034027673304081
Epoch: 7159 Loss: 0.011032463051378727
Epoch: 7160 Loss: 0.011030432768166065
Epoch: 7161 Loss: 0.011027596890926361
Epoch: 7162 Loss: 0.011024746112525463
Epoch: 7163 Loss: 0.011024068109691143
Epoch: 7164 Loss: 0.011020869016647339
Epoch: 7165 Loss: 0.011016905307769775
Epoch: 7166 Loss: 0.01101707573980093
Epoch: 7167 Loss: 0.011014808900654316
Epoch: 7168 Loss: 0.011012312024831772
Epoch: 7169 Loss: 0.011009076610207558
Epoch: 7170 Loss: 0.011005905456840992
Epoch: 7171 Loss: 0.011003865860402584
Epoch: 7172 Loss: 0.011000595055520535
Epoch: 7173 Loss: 0.010998368263244629
Epoch: 7174 Loss: 0.0109969861805439
Epoch: 7175 Loss: 0.01099331583827734
Epoch: 7176 Loss: 0.010993406176567078
Epoch: 7177 Loss: 0.010991229675710201
Epoch: 7178 Loss: 0.010986940003931522
Epoch: 7179 Loss: 0.010985222645103931
Epoch: 7180 Loss: 0.010981207713484764
Epoch: 7181 Loss: 0.010980485938489437
Epoch: 7182 Loss: 0.010978875681757927
Epoch: 7183 Loss: 0.010975086130201817
Epoch: 7184 Loss: 0.010972496122121811
Epoch: 7185 Loss: 0.010970013216137886
Epoch: 7186 Loss: 0.010968695394694805
Epoch: 7187 Loss: 0.010967367328703403
Epoch: 7188 Loss: 0.01096346229314804
Epoch: 7189 Loss: 0.010960306040942669
Epoch: 7190 Loss: 0.010959630832076073
Epoch: 7191 Loss: 0.010958198457956314
Epoch: 7192 Loss: 0.010954922065138817
Epoch: 7193 Loss: 0.010951214469969273
Epoch: 7194 Loss: 0.010949529707431793
Epoch: 7195 Loss: 0.010948947630822659
Epoch: 7196 Loss: 0.010944969952106476
Epoch: 7197 Loss: 0.010942446067929268
Epoch: 7198 Loss: 0.010940039530396461
Epoch: 7199 Loss: 0.010938427411019802
Epoch: 7200 Loss: 0.010934421792626381
Epoch: 7201 Loss: 0.010932505130767822
Epoch: 7202 Loss: 0.010930514894425869
Epoch: 7203 Loss: 0.010929099284112453
Epoch: 7204 Loss: 0.010926051065325737
Epoch: 7205 Loss: 0.010922807268798351
Epoch: 7206 Loss: 0.010921519249677658
Epoch: 7207 Loss: 0.010919945314526558
Epoch: 7208 Loss: 0.010916276834905148
Epoch: 7209 Loss: 0.010914242826402187
Epoch: 7210 Loss: 0.010912124998867512
Epoch: 7211 Loss: 0.01090816780924797
Epoch: 7212 Loss: 0.010908009484410286
Epoch: 7213 Loss: 0.010906286537647247
Epoch: 7214 Loss: 0.0109034338966012
Epoch: 7215 Loss: 0.010900845751166344
Epoch: 7216 Loss: 0.010898093692958355
Epoch: 7217 Loss: 0.0108952010050416
Epoch: 7218 Loss: 0.01089269109070301
Epoch: 7219 Loss: 0.01089041493833065
Epoch: 7220 Loss: 0.010889354161918163
Epoch: 7221 Loss: 0.010885622352361679
Epoch: 7222 Loss: 0.0108827268704772
Epoch: 7223 Loss: 0.010880859568715096
Epoch: 7224 Loss: 0.010879217647016048
Epoch: 7225 Loss: 0.010876622051000595
Epoch: 7226 Loss: 0.010876157321035862
Epoch: 7227 Loss: 0.010872126556932926
Epoch: 7228 Loss: 0.010869123041629791
Epoch: 7229 Loss: 0.010867027565836906
Epoch: 7230 Loss: 0.010866419412195683
Epoch: 7231 Loss: 0.010861733928322792
Epoch: 7232 Loss: 0.010861264541745186
Epoch: 7233 Loss: 0.010856380686163902
Epoch: 7234 Loss: 0.010854500345885754
Epoch: 7235 Loss: 0.010855579748749733
Epoch: 7236 Loss: 0.010850418359041214
Epoch: 7237 Loss: 0.010850497521460056
Epoch: 7238 Loss: 0.010846768505871296
Epoch: 7239 Loss: 0.010844388045370579
Epoch: 7240 Loss: 0.010842039249837399
Epoch: 7241 Loss: 0.010838543064892292
Epoch: 7242 Loss: 0.010838015004992485
Epoch: 7243 Loss: 0.010836600326001644
Epoch: 7244 Loss: 0.010832061059772968
Epoch: 7245 Loss: 0.01083045732229948
Epoch: 7246 Loss: 0.010826797224581242
Epoch: 7247 Loss: 0.010825580917298794
Epoch: 7248 Loss: 0.010823724791407585
Epoch: 7249 Loss: 0.01082139927893877
Epoch: 7250 Loss: 0.010817763395607471
Epoch: 7251 Loss: 0.010816551744937897
Epoch: 7252 Loss: 0.01081410888582468
Epoch: 7253 Loss: 0.01081071887165308
Epoch: 7254 Loss: 0.01081009954214096
Epoch: 7255 Loss: 0.010806215927004814
Epoch: 7256 Loss: 0.010803877376019955
Epoch: 7257 Loss: 0.010802398435771465
Epoch: 7258 Loss: 0.010802038945257664
Epoch: 7259 Loss: 0.010797311551868916
Epoch: 7260 Loss: 0.010795778594911098
Epoch: 7261 Loss: 0.01079373899847269
Epoch: 7262 Loss: 0.010791621170938015
Epoch: 7263 Loss: 0.0107869952917099
Epoch: 7264 Loss: 0.010785968042910099
Epoch: 7265 Loss: 0.010783212259411812
Epoch: 7266 Loss: 0.010782896541059017
Epoch: 7267 Loss: 0.01078020315617323
Epoch: 7268 Loss: 0.010776696726679802
Epoch: 7269 Loss: 0.010775364935398102
Epoch: 7270 Loss: 0.010774357244372368
Epoch: 7271 Loss: 0.010769234038889408
Epoch: 7272 Loss: 0.010766739957034588
Epoch: 7273 Loss: 0.010765382088720798
Epoch: 7274 Loss: 0.010762901045382023
Epoch: 7275 Loss: 0.01076115109026432
Epoch: 7276 Loss: 0.010758780874311924
Epoch: 7277 Loss: 0.010755199939012527
Epoch: 7278 Loss: 0.010753280483186245
Epoch: 7279 Loss: 0.010751355439424515
Epoch: 7280 Loss: 0.01075101550668478
Epoch: 7281 Loss: 0.010748201049864292
Epoch: 7282 Loss: 0.010744205676019192
Epoch: 7283 Loss: 0.010743444785475731
Epoch: 7284 Loss: 0.01074043195694685
Epoch: 7285 Loss: 0.010738971643149853
Epoch: 7286 Loss: 0.010735908523201942
Epoch: 7287 Loss: 0.010733019560575485
Epoch: 7288 Loss: 0.010730015113949776
Epoch: 7289 Loss: 0.010729429312050343
Epoch: 7290 Loss: 0.010726178996264935
Epoch: 7291 Loss: 0.01072658784687519
Epoch: 7292 Loss: 0.0107215391471982
Epoch: 7293 Loss: 0.010720296762883663
Epoch: 7294 Loss: 0.010717328637838364
Epoch: 7295 Loss: 0.01071558054536581
Epoch: 7296 Loss: 0.010712362825870514
Epoch: 7297 Loss: 0.010713769122958183
Epoch: 7298 Loss: 0.010709944181144238
Epoch: 7299 Loss: 0.010706096887588501
Epoch: 7300 Loss: 0.010703803040087223
Epoch: 7301 Loss: 0.010700867511332035
Epoch: 7302 Loss: 0.010699016973376274
Epoch: 7303 Loss: 0.010698629543185234
Epoch: 7304 Loss: 0.010694246739149094
Epoch: 7305 Loss: 0.010693266056478024
Epoch: 7306 Loss: 0.010691637173295021
Epoch: 7307 Loss: 0.01068809162825346
Epoch: 7308 Loss: 0.01068663690239191
Epoch: 7309 Loss: 0.010683060623705387
Epoch: 7310 Loss: 0.010680332779884338
Epoch: 7311 Loss: 0.010678847320377827
Epoch: 7312 Loss: 0.010677477344870567
Epoch: 7313 Loss: 0.010676361620426178
Epoch: 7314 Loss: 0.010673025622963905
Epoch: 7315 Loss: 0.010668984614312649
Epoch: 7316 Loss: 0.01066721323877573
Epoch: 7317 Loss: 0.01066538318991661
Epoch: 7318 Loss: 0.010663008317351341
Epoch: 7319 Loss: 0.01065991260111332
Epoch: 7320 Loss: 0.010659582912921906
Epoch: 7321 Loss: 0.01065676286816597
Epoch: 7322 Loss: 0.010654831305146217
Epoch: 7323 Loss: 0.010651827789843082
Epoch: 7324 Loss: 0.010650078766047955
Epoch: 7325 Loss: 0.01064905896782875
Epoch: 7326 Loss: 0.010644407011568546
Epoch: 7327 Loss: 0.010642397217452526
Epoch: 7328 Loss: 0.010640598833560944
Epoch: 7329 Loss: 0.010637289844453335
Epoch: 7330 Loss: 0.01063755713403225
Epoch: 7331 Loss: 0.010633734986186028
Epoch: 7332 Loss: 0.010631201788783073
Epoch: 7333 Loss: 0.010629279538989067
Epoch: 7334 Loss: 0.010628757998347282
Epoch: 7335 Loss: 0.010624432936310768
Epoch: 7336 Loss: 0.010622608475387096
Epoch: 7337 Loss: 0.010619379580020905
Epoch: 7338 Loss: 0.010617963969707489
Epoch: 7339 Loss: 0.010616726242005825
Epoch: 7340 Loss: 0.010614468716084957
Epoch: 7341 Loss: 0.010610955767333508
Epoch: 7342 Loss: 0.010610240511596203
Epoch: 7343 Loss: 0.010608170181512833
Epoch: 7344 Loss: 0.01060437597334385
Epoch: 7345 Loss: 0.010602821595966816
Epoch: 7346 Loss: 0.01059962622821331
Epoch: 7347 Loss: 0.01060014683753252
Epoch: 7348 Loss: 0.010595827363431454
Epoch: 7349 Loss: 0.010592199862003326
Epoch: 7350 Loss: 0.0105921421200037
Epoch: 7351 Loss: 0.010590337216854095
Epoch: 7352 Loss: 0.010586517862975597
Epoch: 7353 Loss: 0.010583959519863129
Epoch: 7354 Loss: 0.010583252646028996
Epoch: 7355 Loss: 0.010580700822174549
Epoch: 7356 Loss: 0.010578091256320477
Epoch: 7357 Loss: 0.010576060973107815
Epoch: 7358 Loss: 0.0105743408203125
Epoch: 7359 Loss: 0.010570857673883438
Epoch: 7360 Loss: 0.010569646023213863
Epoch: 7361 Loss: 0.010567990131676197
Epoch: 7362 Loss: 0.010565688833594322
Epoch: 7363 Loss: 0.010563683696091175
Epoch: 7364 Loss: 0.010560214519500732
Epoch: 7365 Loss: 0.010557575151324272
Epoch: 7366 Loss: 0.010556250810623169
Epoch: 7367 Loss: 0.010554550215601921
Epoch: 7368 Loss: 0.01055179350078106
Epoch: 7369 Loss: 0.010549047961831093
Epoch: 7370 Loss: 0.010546882636845112
Epoch: 7371 Loss: 0.010544707998633385
Epoch: 7372 Loss: 0.010542397387325764
Epoch: 7373 Loss: 0.010541008785367012
Epoch: 7374 Loss: 0.010538325645029545
Epoch: 7375 Loss: 0.010535265319049358
Epoch: 7376 Loss: 0.010533963330090046
Epoch: 7377 Loss: 0.010531975887715816
Epoch: 7378 Loss: 0.010530313476920128
Epoch: 7379 Loss: 0.010527371428906918
Epoch: 7380 Loss: 0.010524912737309933
Epoch: 7381 Loss: 0.010522546246647835
Epoch: 7382 Loss: 0.010521089658141136
Epoch: 7383 Loss: 0.010517755523324013
Epoch: 7384 Loss: 0.010515761561691761
Epoch: 7385 Loss: 0.010514169000089169
Epoch: 7386 Loss: 0.01051081158220768
Epoch: 7387 Loss: 0.010509487241506577
Epoch: 7388 Loss: 0.01050989143550396
Epoch: 7389 Loss: 0.010505345650017262
Epoch: 7390 Loss: 0.010504336096346378
Epoch: 7391 Loss: 0.010500429198145866
Epoch: 7392 Loss: 0.010499255731701851
Epoch: 7393 Loss: 0.010497420094907284
Epoch: 7394 Loss: 0.010494323447346687
Epoch: 7395 Loss: 0.010492298752069473
Epoch: 7396 Loss: 0.010490807704627514
Epoch: 7397 Loss: 0.01048800814896822
Epoch: 7398 Loss: 0.010486321523785591
Epoch: 7399 Loss: 0.010483626276254654
Epoch: 7400 Loss: 0.010481448844075203
Epoch: 7401 Loss: 0.010479834862053394
Epoch: 7402 Loss: 0.010476759634912014
Epoch: 7403 Loss: 0.010475648567080498
Epoch: 7404 Loss: 0.010473117232322693
Epoch: 7405 Loss: 0.010470062494277954
Epoch: 7406 Loss: 0.010468089953064919
Epoch: 7407 Loss: 0.010465918108820915
Epoch: 7408 Loss: 0.01046349760144949
Epoch: 7409 Loss: 0.010462555103003979
Epoch: 7410 Loss: 0.010459755547344685
Epoch: 7411 Loss: 0.010456819087266922
Epoch: 7412 Loss: 0.010454400442540646
Epoch: 7413 Loss: 0.010453250259160995
Epoch: 7414 Loss: 0.01045183278620243
Epoch: 7415 Loss: 0.010447592474520206
Epoch: 7416 Loss: 0.010446247644722462
Epoch: 7417 Loss: 0.010444954968988895
Epoch: 7418 Loss: 0.010443711653351784
Epoch: 7419 Loss: 0.010439486242830753
Epoch: 7420 Loss: 0.010437472723424435
Epoch: 7421 Loss: 0.010434839874505997
Epoch: 7422 Loss: 0.010432343930006027
Epoch: 7423 Loss: 0.010431493632495403
Epoch: 7424 Loss: 0.01042964681982994
Epoch: 7425 Loss: 0.010426843538880348
Epoch: 7426 Loss: 0.01042430941015482
Epoch: 7427 Loss: 0.010422440245747566
Epoch: 7428 Loss: 0.010420430451631546
Epoch: 7429 Loss: 0.010418656282126904
Epoch: 7430 Loss: 0.010415495373308659
Epoch: 7431 Loss: 0.010415700264275074
Epoch: 7432 Loss: 0.010412370786070824
Epoch: 7433 Loss: 0.010410650633275509
Epoch: 7434 Loss: 0.010408082976937294
Epoch: 7435 Loss: 0.010405121371150017
Epoch: 7436 Loss: 0.010403807274997234
Epoch: 7437 Loss: 0.010401285253465176
Epoch: 7438 Loss: 0.01039829757064581
Epoch: 7439 Loss: 0.010397111065685749
Epoch: 7440 Loss: 0.010395215824246407
Epoch: 7441 Loss: 0.010391478426754475
Epoch: 7442 Loss: 0.010390001349151134
Epoch: 7443 Loss: 0.010388010181486607
Epoch: 7444 Loss: 0.010385427623987198
Epoch: 7445 Loss: 0.010385540314018726
Epoch: 7446 Loss: 0.0103820925578475
Epoch: 7447 Loss: 0.010379599407315254
Epoch: 7448 Loss: 0.010377848520874977
Epoch: 7449 Loss: 0.010374707169830799
Epoch: 7450 Loss: 0.010372555814683437
Epoch: 7451 Loss: 0.01037200354039669
Epoch: 7452 Loss: 0.010369446128606796
Epoch: 7453 Loss: 0.010365044698119164
Epoch: 7454 Loss: 0.010364667512476444
Epoch: 7455 Loss: 0.010361806489527225
Epoch: 7456 Loss: 0.010360680520534515
Epoch: 7457 Loss: 0.010357682593166828
Epoch: 7458 Loss: 0.010356191545724869
Epoch: 7459 Loss: 0.010353604331612587
Epoch: 7460 Loss: 0.010350977070629597
Epoch: 7461 Loss: 0.010350102558732033
Epoch: 7462 Loss: 0.010347390547394753
Epoch: 7463 Loss: 0.010345560498535633
Epoch: 7464 Loss: 0.010343671776354313
Epoch: 7465 Loss: 0.010342667810618877
Epoch: 7466 Loss: 0.010338193736970425
Epoch: 7467 Loss: 0.010336147621273994
Epoch: 7468 Loss: 0.010333661921322346
Epoch: 7469 Loss: 0.010332143865525723
Epoch: 7470 Loss: 0.01033040415495634
Epoch: 7471 Loss: 0.010328918695449829
Epoch: 7472 Loss: 0.01032544020563364
Epoch: 7473 Loss: 0.010323834605515003
Epoch: 7474 Loss: 0.01032155193388462
Epoch: 7475 Loss: 0.01031953003257513
Epoch: 7476 Loss: 0.01031732652336359
Epoch: 7477 Loss: 0.010316165164113045
Epoch: 7478 Loss: 0.010313563048839569
Epoch: 7479 Loss: 0.010312293656170368
Epoch: 7480 Loss: 0.01030801422894001
Epoch: 7481 Loss: 0.01030774600803852
Epoch: 7482 Loss: 0.010304631665349007
Epoch: 7483 Loss: 0.01030445471405983
Epoch: 7484 Loss: 0.0103001669049263
Epoch: 7485 Loss: 0.010298321023583412
Epoch: 7486 Loss: 0.010297171771526337
Epoch: 7487 Loss: 0.010295138694345951
Epoch: 7488 Loss: 0.010292083956301212
Epoch: 7489 Loss: 0.010290543548762798
Epoch: 7490 Loss: 0.010287360288202763
Epoch: 7491 Loss: 0.010286089032888412
Epoch: 7492 Loss: 0.01028228085488081
Epoch: 7493 Loss: 0.010280653834342957
Epoch: 7494 Loss: 0.010280370712280273
Epoch: 7495 Loss: 0.010277924127876759
Epoch: 7496 Loss: 0.010276143439114094
Epoch: 7497 Loss: 0.010272283107042313
Epoch: 7498 Loss: 0.010270573198795319
Epoch: 7499 Loss: 0.010267489589750767
Epoch: 7500 Loss: 0.010267561301589012
Epoch: 7501 Loss: 0.010265425778925419
Epoch: 7502 Loss: 0.010262792930006981
Epoch: 7503 Loss: 0.010259903036057949
Epoch: 7504 Loss: 0.010257771238684654
Epoch: 7505 Loss: 0.010255980305373669
Epoch: 7506 Loss: 0.01025304477661848
Epoch: 7507 Loss: 0.010251960717141628
Epoch: 7508 Loss: 0.010249141603708267
Epoch: 7509 Loss: 0.010247420519590378
Epoch: 7510 Loss: 0.010246360674500465
Epoch: 7511 Loss: 0.01024345401674509
Epoch: 7512 Loss: 0.010243207216262817
Epoch: 7513 Loss: 0.01023900881409645
Epoch: 7514 Loss: 0.01023646630346775
Epoch: 7515 Loss: 0.010235647670924664
Epoch: 7516 Loss: 0.010233157314360142
Epoch: 7517 Loss: 0.010231266729533672
Epoch: 7518 Loss: 0.010229941457509995
Epoch: 7519 Loss: 0.010227479971945286
Epoch: 7520 Loss: 0.01022548507899046
Epoch: 7521 Loss: 0.010221634991466999
Epoch: 7522 Loss: 0.010221649892628193
Epoch: 7523 Loss: 0.010217788629233837
Epoch: 7524 Loss: 0.010216244496405125
Epoch: 7525 Loss: 0.010214405134320259
Epoch: 7526 Loss: 0.010212458670139313
Epoch: 7527 Loss: 0.01021130196750164
Epoch: 7528 Loss: 0.010208525694906712
Epoch: 7529 Loss: 0.010205497033894062
Epoch: 7530 Loss: 0.010204226709902287
Epoch: 7531 Loss: 0.010201869532465935
Epoch: 7532 Loss: 0.010200663469731808
Epoch: 7533 Loss: 0.010196835733950138
Epoch: 7534 Loss: 0.01019531674683094
Epoch: 7535 Loss: 0.010194111615419388
Epoch: 7536 Loss: 0.01019164640456438
Epoch: 7537 Loss: 0.010188820771872997
Epoch: 7538 Loss: 0.010186998173594475
Epoch: 7539 Loss: 0.010185194201767445
Epoch: 7540 Loss: 0.010182426311075687
Epoch: 7541 Loss: 0.010179981589317322
Epoch: 7542 Loss: 0.010178731754422188
Epoch: 7543 Loss: 0.01017758622765541
Epoch: 7544 Loss: 0.010174544528126717
Epoch: 7545 Loss: 0.010171695612370968
Epoch: 7546 Loss: 0.01017136499285698
Epoch: 7547 Loss: 0.01016843132674694
Epoch: 7548 Loss: 0.010166093707084656
Epoch: 7549 Loss: 0.010165217332541943
Epoch: 7550 Loss: 0.01016223058104515
Epoch: 7551 Loss: 0.01016122754663229
Epoch: 7552 Loss: 0.010157736018300056
Epoch: 7553 Loss: 0.010157465003430843
Epoch: 7554 Loss: 0.01015425194054842
Epoch: 7555 Loss: 0.010152081958949566
Epoch: 7556 Loss: 0.010149001143872738
Epoch: 7557 Loss: 0.010148373432457447
Epoch: 7558 Loss: 0.01014622300863266
Epoch: 7559 Loss: 0.010144077241420746
Epoch: 7560 Loss: 0.010142723098397255
Epoch: 7561 Loss: 0.010139676742255688
Epoch: 7562 Loss: 0.010138172656297684
Epoch: 7563 Loss: 0.010135146789252758
Epoch: 7564 Loss: 0.010133334435522556
Epoch: 7565 Loss: 0.010130808688700199
Epoch: 7566 Loss: 0.010129095055162907
Epoch: 7567 Loss: 0.010126509703695774
Epoch: 7568 Loss: 0.010124338790774345
Epoch: 7569 Loss: 0.010123448446393013
Epoch: 7570 Loss: 0.010121015831828117
Epoch: 7571 Loss: 0.010120082646608353
Epoch: 7572 Loss: 0.010117077268660069
Epoch: 7573 Loss: 0.010115613229572773
Epoch: 7574 Loss: 0.01011350005865097
Epoch: 7575 Loss: 0.010110202245414257
Epoch: 7576 Loss: 0.01010812260210514
Epoch: 7577 Loss: 0.010106758214533329
Epoch: 7578 Loss: 0.010104869492352009
Epoch: 7579 Loss: 0.010102171450853348
Epoch: 7580 Loss: 0.010099968872964382
Epoch: 7581 Loss: 0.010099345818161964
Epoch: 7582 Loss: 0.010096448473632336
Epoch: 7583 Loss: 0.01009486522525549
Epoch: 7584 Loss: 0.01009197998791933
Epoch: 7585 Loss: 0.01009165495634079
Epoch: 7586 Loss: 0.010087224654853344
Epoch: 7587 Loss: 0.010085675865411758
Epoch: 7588 Loss: 0.010084121488034725
Epoch: 7589 Loss: 0.010081690736114979
Epoch: 7590 Loss: 0.010079802945256233
Epoch: 7591 Loss: 0.01007893867790699
Epoch: 7592 Loss: 0.010074904188513756
Epoch: 7593 Loss: 0.010073636658489704
Epoch: 7594 Loss: 0.010071256197988987
Epoch: 7595 Loss: 0.01006882544606924
Epoch: 7596 Loss: 0.010067169554531574
Epoch: 7597 Loss: 0.01006479375064373
Epoch: 7598 Loss: 0.010062566958367825
Epoch: 7599 Loss: 0.010062801651656628
Epoch: 7600 Loss: 0.010060440748929977
Epoch: 7601 Loss: 0.01005720067769289
Epoch: 7602 Loss: 0.010056122206151485
Epoch: 7603 Loss: 0.010053769685328007
Epoch: 7604 Loss: 0.010051054880023003
Epoch: 7605 Loss: 0.010048952884972095
Epoch: 7606 Loss: 0.010047239251434803
Epoch: 7607 Loss: 0.010044680908322334
Epoch: 7608 Loss: 0.010043672285974026
Epoch: 7609 Loss: 0.010039524175226688
Epoch: 7610 Loss: 0.010039440356194973
Epoch: 7611 Loss: 0.01003839261829853
Epoch: 7612 Loss: 0.010036264546215534
Epoch: 7613 Loss: 0.010033170692622662
Epoch: 7614 Loss: 0.01003124751150608
Epoch: 7615 Loss: 0.010029003024101257
Epoch: 7616 Loss: 0.010027221404016018
Epoch: 7617 Loss: 0.010025717318058014
Epoch: 7618 Loss: 0.010022223927080631
Epoch: 7619 Loss: 0.010020404122769833
Epoch: 7620 Loss: 0.010020640678703785
Epoch: 7621 Loss: 0.010017089545726776
Epoch: 7622 Loss: 0.010014928877353668
Epoch: 7623 Loss: 0.010013618506491184
Epoch: 7624 Loss: 0.010010845959186554
Epoch: 7625 Loss: 0.010009015910327435
Epoch: 7626 Loss: 0.010006655007600784
Epoch: 7627 Loss: 0.010004427284002304
Epoch: 7628 Loss: 0.01000187173485756
Epoch: 7629 Loss: 0.0099997753277421
Epoch: 7630 Loss: 0.009999413043260574
Epoch: 7631 Loss: 0.00999754574149847
Epoch: 7632 Loss: 0.00999387539923191
Epoch: 7633 Loss: 0.009992633014917374
Epoch: 7634 Loss: 0.0099889962002635
Epoch: 7635 Loss: 0.009989246726036072
Epoch: 7636 Loss: 0.00998726487159729
Epoch: 7637 Loss: 0.009984618984162807
Epoch: 7638 Loss: 0.009981905110180378
Epoch: 7639 Loss: 0.009981128387153149
Epoch: 7640 Loss: 0.009977192617952824
Epoch: 7641 Loss: 0.009976109489798546
Epoch: 7642 Loss: 0.009974650107324123
Epoch: 7643 Loss: 0.009973233565688133
Epoch: 7644 Loss: 0.009969527833163738
Epoch: 7645 Loss: 0.009967860765755177
Epoch: 7646 Loss: 0.009966173209249973
Epoch: 7647 Loss: 0.009965656325221062
Epoch: 7648 Loss: 0.009962087497115135
Epoch: 7649 Loss: 0.009960144758224487
Epoch: 7650 Loss: 0.009959313087165356
Epoch: 7651 Loss: 0.00995715707540512
Epoch: 7652 Loss: 0.009955648332834244
Epoch: 7653 Loss: 0.009952363558113575
Epoch: 7654 Loss: 0.009950194507837296
Epoch: 7655 Loss: 0.00994901917874813
Epoch: 7656 Loss: 0.009945766068994999
Epoch: 7657 Loss: 0.009944298304617405
Epoch: 7658 Loss: 0.009943610057234764
Epoch: 7659 Loss: 0.009940288960933685
Epoch: 7660 Loss: 0.009939111769199371
Epoch: 7661 Loss: 0.009936464950442314
Epoch: 7662 Loss: 0.00993506982922554
Epoch: 7663 Loss: 0.009931995533406734
Epoch: 7664 Loss: 0.009931053966283798
Epoch: 7665 Loss: 0.009928970597684383
Epoch: 7666 Loss: 0.009926543571054935
Epoch: 7667 Loss: 0.00992413330823183
Epoch: 7668 Loss: 0.009922121651470661
Epoch: 7669 Loss: 0.009920872747898102
Epoch: 7670 Loss: 0.009918401017785072
Epoch: 7671 Loss: 0.009917370975017548
Epoch: 7672 Loss: 0.00991537980735302
Epoch: 7673 Loss: 0.009911715053021908
Epoch: 7674 Loss: 0.009909918531775475
Epoch: 7675 Loss: 0.009908381849527359
Epoch: 7676 Loss: 0.009907756932079792
Epoch: 7677 Loss: 0.009905447252094746
Epoch: 7678 Loss: 0.009902216494083405
Epoch: 7679 Loss: 0.009900965727865696
Epoch: 7680 Loss: 0.009898421354591846
Epoch: 7681 Loss: 0.009897521696984768
Epoch: 7682 Loss: 0.009895325638353825
Epoch: 7683 Loss: 0.009892805479466915
Epoch: 7684 Loss: 0.009890779852867126
Epoch: 7685 Loss: 0.009888207539916039
Epoch: 7686 Loss: 0.009886658750474453
Epoch: 7687 Loss: 0.009884651750326157
Epoch: 7688 Loss: 0.009882581420242786
Epoch: 7689 Loss: 0.009880894795060158
Epoch: 7690 Loss: 0.009881730191409588
Epoch: 7691 Loss: 0.009877988137304783
Epoch: 7692 Loss: 0.009874898940324783
Epoch: 7693 Loss: 0.009873329661786556
Epoch: 7694 Loss: 0.009871631860733032
Epoch: 7695 Loss: 0.009869403205811977
Epoch: 7696 Loss: 0.009868156164884567
Epoch: 7697 Loss: 0.009865106083452702
Epoch: 7698 Loss: 0.00986337848007679
Epoch: 7699 Loss: 0.00986053142696619
Epoch: 7700 Loss: 0.009860950522124767
Epoch: 7701 Loss: 0.00985661894083023
Epoch: 7702 Loss: 0.009856991469860077
Epoch: 7703 Loss: 0.009853936731815338
Epoch: 7704 Loss: 0.009850527159869671
Epoch: 7705 Loss: 0.009849377907812595
Epoch: 7706 Loss: 0.009847233071923256
Epoch: 7707 Loss: 0.009845474734902382
Epoch: 7708 Loss: 0.00984531082212925
Epoch: 7709 Loss: 0.009842834435403347
Epoch: 7710 Loss: 0.009839650243520737
Epoch: 7711 Loss: 0.009837272576987743
Epoch: 7712 Loss: 0.009835518896579742
Epoch: 7713 Loss: 0.00983328279107809
Epoch: 7714 Loss: 0.009832285344600677
Epoch: 7715 Loss: 0.009830789640545845
Epoch: 7716 Loss: 0.009827717207372189
Epoch: 7717 Loss: 0.009827254340052605
Epoch: 7718 Loss: 0.009824328124523163
Epoch: 7719 Loss: 0.009822303429245949
Epoch: 7720 Loss: 0.009820429608225822
Epoch: 7721 Loss: 0.009818191640079021
Epoch: 7722 Loss: 0.009815562516450882
Epoch: 7723 Loss: 0.009813647717237473
Epoch: 7724 Loss: 0.009813622571527958
Epoch: 7725 Loss: 0.009810484945774078
Epoch: 7726 Loss: 0.009808803908526897
Epoch: 7727 Loss: 0.009806390851736069
Epoch: 7728 Loss: 0.009804611094295979
Epoch: 7729 Loss: 0.009801959618926048
Epoch: 7730 Loss: 0.00980212353169918
Epoch: 7731 Loss: 0.00979922991245985
Epoch: 7732 Loss: 0.009796013124287128
Epoch: 7733 Loss: 0.009794744662940502
Epoch: 7734 Loss: 0.009793520905077457
Epoch: 7735 Loss: 0.00979196559637785
Epoch: 7736 Loss: 0.009789974428713322
Epoch: 7737 Loss: 0.00978722795844078
Epoch: 7738 Loss: 0.009784440509974957
Epoch: 7739 Loss: 0.009782969020307064
Epoch: 7740 Loss: 0.009781170636415482
Epoch: 7741 Loss: 0.009778918698430061
Epoch: 7742 Loss: 0.009778115898370743
Epoch: 7743 Loss: 0.009775589220225811
Epoch: 7744 Loss: 0.009773166850209236
Epoch: 7745 Loss: 0.009772061370313168
Epoch: 7746 Loss: 0.009769565425813198
Epoch: 7747 Loss: 0.009767113253474236
Epoch: 7748 Loss: 0.009765883907675743
Epoch: 7749 Loss: 0.009763858281075954
Epoch: 7750 Loss: 0.009762129746377468
Epoch: 7751 Loss: 0.009759865701198578
Epoch: 7752 Loss: 0.009757633320987225
Epoch: 7753 Loss: 0.009755955077707767
Epoch: 7754 Loss: 0.009753473103046417
Epoch: 7755 Loss: 0.009752498008310795
Epoch: 7756 Loss: 0.009750417433679104
Epoch: 7757 Loss: 0.009748492389917374
Epoch: 7758 Loss: 0.009746210649609566
Epoch: 7759 Loss: 0.009743895381689072
Epoch: 7760 Loss: 0.009742936119437218
Epoch: 7761 Loss: 0.009742333553731441
Epoch: 7762 Loss: 0.009737702086567879
Epoch: 7763 Loss: 0.009736375883221626
Epoch: 7764 Loss: 0.00973572302609682
Epoch: 7765 Loss: 0.00973392091691494
Epoch: 7766 Loss: 0.009731377474963665
Epoch: 7767 Loss: 0.009728312492370605
Epoch: 7768 Loss: 0.009726062417030334
Epoch: 7769 Loss: 0.00972504448145628
Epoch: 7770 Loss: 0.009723808616399765
Epoch: 7771 Loss: 0.009720821864902973
Epoch: 7772 Loss: 0.009719552472233772
Epoch: 7773 Loss: 0.009718118235468864
Epoch: 7774 Loss: 0.009716371074318886
Epoch: 7775 Loss: 0.00971401110291481
Epoch: 7776 Loss: 0.009710600599646568
Epoch: 7777 Loss: 0.009709932841360569
Epoch: 7778 Loss: 0.00970732793211937
Epoch: 7779 Loss: 0.009705537930130959
Epoch: 7780 Loss: 0.009705448523163795
Epoch: 7781 Loss: 0.009700849652290344
Epoch: 7782 Loss: 0.009699677117168903
Epoch: 7783 Loss: 0.009698017500340939
Epoch: 7784 Loss: 0.009698046371340752
Epoch: 7785 Loss: 0.009695260785520077
Epoch: 7786 Loss: 0.009691852144896984
Epoch: 7787 Loss: 0.009690839797258377
Epoch: 7788 Loss: 0.009687383659183979
Epoch: 7789 Loss: 0.009686735458672047
Epoch: 7790 Loss: 0.009684440679848194
Epoch: 7791 Loss: 0.009681782685220242
Epoch: 7792 Loss: 0.009682276286184788
Epoch: 7793 Loss: 0.009679580107331276
Epoch: 7794 Loss: 0.009676485322415829
Epoch: 7795 Loss: 0.009676075540482998
Epoch: 7796 Loss: 0.009672729298472404
Epoch: 7797 Loss: 0.009671662002801895
Epoch: 7798 Loss: 0.009670081548392773
Epoch: 7799 Loss: 0.009669052436947823
Epoch: 7800 Loss: 0.00966603308916092
Epoch: 7801 Loss: 0.009663532488048077
Epoch: 7802 Loss: 0.009662395343184471
Epoch: 7803 Loss: 0.009660367853939533
Epoch: 7804 Loss: 0.009657875634729862
Epoch: 7805 Loss: 0.009657379239797592
Epoch: 7806 Loss: 0.00965373869985342
Epoch: 7807 Loss: 0.009653459303081036
Epoch: 7808 Loss: 0.009650682099163532
Epoch: 7809 Loss: 0.009648329578340054
Epoch: 7810 Loss: 0.009648634120821953
Epoch: 7811 Loss: 0.00964551605284214
Epoch: 7812 Loss: 0.00964248925447464
Epoch: 7813 Loss: 0.009641168639063835
Epoch: 7814 Loss: 0.009640023112297058
Epoch: 7815 Loss: 0.009637515060603619
Epoch: 7816 Loss: 0.009634731337428093
Epoch: 7817 Loss: 0.009634526446461678
Epoch: 7818 Loss: 0.00963262002915144
Epoch: 7819 Loss: 0.009629548527300358
Epoch: 7820 Loss: 0.009628472849726677
Epoch: 7821 Loss: 0.009625613689422607
Epoch: 7822 Loss: 0.009623659774661064
Epoch: 7823 Loss: 0.009622573852539062
Epoch: 7824 Loss: 0.009619707241654396
Epoch: 7825 Loss: 0.009618759155273438
Epoch: 7826 Loss: 0.009617486037313938
Epoch: 7827 Loss: 0.009613808244466782
Epoch: 7828 Loss: 0.009613524191081524
Epoch: 7829 Loss: 0.00961063802242279
Epoch: 7830 Loss: 0.009608987718820572
Epoch: 7831 Loss: 0.00960894487798214
Epoch: 7832 Loss: 0.0096055306494236
Epoch: 7833 Loss: 0.00960275623947382
Epoch: 7834 Loss: 0.009602070786058903
Epoch: 7835 Loss: 0.009599990211427212
Epoch: 7836 Loss: 0.009598412550985813
Epoch: 7837 Loss: 0.00959536712616682
Epoch: 7838 Loss: 0.009593273513019085
Epoch: 7839 Loss: 0.0095908772200346
Epoch: 7840 Loss: 0.009589891880750656
Epoch: 7841 Loss: 0.009589377790689468
Epoch: 7842 Loss: 0.009585533291101456
Epoch: 7843 Loss: 0.009584791027009487
Epoch: 7844 Loss: 0.009581858292222023
Epoch: 7845 Loss: 0.009580336511135101
Epoch: 7846 Loss: 0.009579396806657314
Epoch: 7847 Loss: 0.009576685726642609
Epoch: 7848 Loss: 0.00957509409636259
Epoch: 7849 Loss: 0.009573196060955524
Epoch: 7850 Loss: 0.00957280583679676
Epoch: 7851 Loss: 0.009569128043949604
Epoch: 7852 Loss: 0.009567022323608398
Epoch: 7853 Loss: 0.009566988795995712
Epoch: 7854 Loss: 0.009564332664012909
Epoch: 7855 Loss: 0.009561783634126186
Epoch: 7856 Loss: 0.009559964761137962
Epoch: 7857 Loss: 0.009558547288179398
Epoch: 7858 Loss: 0.009556766599416733
Epoch: 7859 Loss: 0.009554318152368069
Epoch: 7860 Loss: 0.009552747942507267
Epoch: 7861 Loss: 0.009551242925226688
Epoch: 7862 Loss: 0.009548458270728588
Epoch: 7863 Loss: 0.00954760704189539
Epoch: 7864 Loss: 0.009546174667775631
Epoch: 7865 Loss: 0.009543269872665405
Epoch: 7866 Loss: 0.009541152976453304
Epoch: 7867 Loss: 0.009540664032101631
Epoch: 7868 Loss: 0.009538177400827408
Epoch: 7869 Loss: 0.009535602293908596
Epoch: 7870 Loss: 0.00953388400375843
Epoch: 7871 Loss: 0.009532216936349869
Epoch: 7872 Loss: 0.009529606439173222
Epoch: 7873 Loss: 0.00952918641269207
Epoch: 7874 Loss: 0.00952545739710331
Epoch: 7875 Loss: 0.009525053203105927
Epoch: 7876 Loss: 0.00952437985688448
Epoch: 7877 Loss: 0.009521834552288055
Epoch: 7878 Loss: 0.009519125334918499
Epoch: 7879 Loss: 0.009518829174339771
Epoch: 7880 Loss: 0.009515151381492615
Epoch: 7881 Loss: 0.009513525292277336
Epoch: 7882 Loss: 0.009511756710708141
Epoch: 7883 Loss: 0.009509824216365814
Epoch: 7884 Loss: 0.0095093147829175
Epoch: 7885 Loss: 0.009506218135356903
Epoch: 7886 Loss: 0.009505340829491615
Epoch: 7887 Loss: 0.009502525441348553
Epoch: 7888 Loss: 0.009501506574451923
Epoch: 7889 Loss: 0.00949794240295887
Epoch: 7890 Loss: 0.009497353807091713
Epoch: 7891 Loss: 0.009494494646787643
Epoch: 7892 Loss: 0.009494256228208542
Epoch: 7893 Loss: 0.009491520002484322
Epoch: 7894 Loss: 0.009489906020462513
Epoch: 7895 Loss: 0.009488306008279324
Epoch: 7896 Loss: 0.009487262926995754
Epoch: 7897 Loss: 0.009485182352364063
Epoch: 7898 Loss: 0.009481211192905903
Epoch: 7899 Loss: 0.009482262656092644
Epoch: 7900 Loss: 0.009478875435888767
Epoch: 7901 Loss: 0.009477278217673302
Epoch: 7902 Loss: 0.009473646059632301
Epoch: 7903 Loss: 0.0094714080914855
Epoch: 7904 Loss: 0.009472013451159
Epoch: 7905 Loss: 0.009470894932746887
Epoch: 7906 Loss: 0.009467930532991886
Epoch: 7907 Loss: 0.009465280920267105
Epoch: 7908 Loss: 0.009464582428336143
Epoch: 7909 Loss: 0.009461566805839539
Epoch: 7910 Loss: 0.009460847824811935
Epoch: 7911 Loss: 0.009458179585635662
Epoch: 7912 Loss: 0.009456891566514969
Epoch: 7913 Loss: 0.009454688057303429
Epoch: 7914 Loss: 0.00945242028683424
Epoch: 7915 Loss: 0.009451191872358322
Epoch: 7916 Loss: 0.009449245408177376
Epoch: 7917 Loss: 0.009447844699025154
Epoch: 7918 Loss: 0.00944634061306715
Epoch: 7919 Loss: 0.00944322906434536
Epoch: 7920 Loss: 0.009440865367650986
Epoch: 7921 Loss: 0.009440875612199306
Epoch: 7922 Loss: 0.009437673725187778
Epoch: 7923 Loss: 0.00943791400641203
Epoch: 7924 Loss: 0.009435221552848816
Epoch: 7925 Loss: 0.009432314895093441
Epoch: 7926 Loss: 0.009431208483874798
Epoch: 7927 Loss: 0.009428919292986393
Epoch: 7928 Loss: 0.00942723173648119
Epoch: 7929 Loss: 0.00942511297762394
Epoch: 7930 Loss: 0.00942461472004652
Epoch: 7931 Loss: 0.009421678259968758
Epoch: 7932 Loss: 0.009419577196240425
Epoch: 7933 Loss: 0.009418537840247154
Epoch: 7934 Loss: 0.009417666122317314
Epoch: 7935 Loss: 0.00941373035311699
Epoch: 7936 Loss: 0.009411733597517014
Epoch: 7937 Loss: 0.009411179460585117
Epoch: 7938 Loss: 0.009408675134181976
Epoch: 7939 Loss: 0.00940635520964861
Epoch: 7940 Loss: 0.009406556375324726
Epoch: 7941 Loss: 0.009403079748153687
Epoch: 7942 Loss: 0.009401947259902954
Epoch: 7943 Loss: 0.00939927902072668
Epoch: 7944 Loss: 0.009397587738931179
Epoch: 7945 Loss: 0.00939688179641962
Epoch: 7946 Loss: 0.009396337904036045
Epoch: 7947 Loss: 0.00939193181693554
Epoch: 7948 Loss: 0.00939054787158966
Epoch: 7949 Loss: 0.009389188140630722
Epoch: 7950 Loss: 0.009387084282934666
Epoch: 7951 Loss: 0.009385108016431332
Epoch: 7952 Loss: 0.009383208118379116
Epoch: 7953 Loss: 0.009381501004099846
Epoch: 7954 Loss: 0.009379609487950802
Epoch: 7955 Loss: 0.00937776081264019
Epoch: 7956 Loss: 0.009376793168485165
Epoch: 7957 Loss: 0.009373720735311508
Epoch: 7958 Loss: 0.009373117238283157
Epoch: 7959 Loss: 0.009369983337819576
Epoch: 7960 Loss: 0.00936906784772873
Epoch: 7961 Loss: 0.009367558173835278
Epoch: 7962 Loss: 0.009365847334265709
Epoch: 7963 Loss: 0.009362651035189629
Epoch: 7964 Loss: 0.009361491538584232
Epoch: 7965 Loss: 0.009360224939882755
Epoch: 7966 Loss: 0.00935821421444416
Epoch: 7967 Loss: 0.009355966933071613
Epoch: 7968 Loss: 0.009354252368211746
Epoch: 7969 Loss: 0.009352500550448895
Epoch: 7970 Loss: 0.009350324049592018
Epoch: 7971 Loss: 0.00934973731637001
Epoch: 7972 Loss: 0.00934695079922676
Epoch: 7973 Loss: 0.009345419704914093
Epoch: 7974 Loss: 0.009343842975795269
Epoch: 7975 Loss: 0.009343128651380539
Epoch: 7976 Loss: 0.009339937008917332
Epoch: 7977 Loss: 0.009337909519672394
Epoch: 7978 Loss: 0.009337681345641613
Epoch: 7979 Loss: 0.009334765374660492
Epoch: 7980 Loss: 0.009333593770861626
Epoch: 7981 Loss: 0.009331179782748222
Epoch: 7982 Loss: 0.009328561834990978
Epoch: 7983 Loss: 0.009327295236289501
Epoch: 7984 Loss: 0.009325923398137093
Epoch: 7985 Loss: 0.009324649348855019
Epoch: 7986 Loss: 0.009322142228484154
Epoch: 7987 Loss: 0.009319505654275417
Epoch: 7988 Loss: 0.009318160824477673
Epoch: 7989 Loss: 0.009318300522863865
Epoch: 7990 Loss: 0.009314515627920628
Epoch: 7991 Loss: 0.009313149377703667
Epoch: 7992 Loss: 0.009310834109783173
Epoch: 7993 Loss: 0.009310240857303143
Epoch: 7994 Loss: 0.009308257140219212
Epoch: 7995 Loss: 0.009306867606937885
Epoch: 7996 Loss: 0.009304342791438103
Epoch: 7997 Loss: 0.009302610531449318
Epoch: 7998 Loss: 0.009299946017563343
Epoch: 7999 Loss: 0.009299523197114468
Epoch: 8000 Loss: 0.00929664634168148
Epoch: 8001 Loss: 0.009296243079006672
Epoch: 8002 Loss: 0.009293786250054836
Epoch: 8003 Loss: 0.009292918257415295
Epoch: 8004 Loss: 0.009289873763918877
Epoch: 8005 Loss: 0.009288093075156212
Epoch: 8006 Loss: 0.009287625551223755
Epoch: 8007 Loss: 0.009286565706133842
Epoch: 8008 Loss: 0.009283775463700294
Epoch: 8009 Loss: 0.009281228296458721
Epoch: 8010 Loss: 0.009278998710215092
Epoch: 8011 Loss: 0.009278110228478909
Epoch: 8012 Loss: 0.009275881573557854
Epoch: 8013 Loss: 0.00927436351776123
Epoch: 8014 Loss: 0.009271964430809021
Epoch: 8015 Loss: 0.009269952774047852
Epoch: 8016 Loss: 0.009269981645047665
Epoch: 8017 Loss: 0.00926816277205944
Epoch: 8018 Loss: 0.009264382533729076
Epoch: 8019 Loss: 0.009263196960091591
Epoch: 8020 Loss: 0.009261761792004108
Epoch: 8021 Loss: 0.009259912185370922
Epoch: 8022 Loss: 0.009258401580154896
Epoch: 8023 Loss: 0.009256146848201752
Epoch: 8024 Loss: 0.009254149161279202
Epoch: 8025 Loss: 0.009252750314772129
Epoch: 8026 Loss: 0.009251761250197887
Epoch: 8027 Loss: 0.009248767048120499
Epoch: 8028 Loss: 0.009247669950127602
Epoch: 8029 Loss: 0.00924686063081026
Epoch: 8030 Loss: 0.009243111126124859
Epoch: 8031 Loss: 0.009242186322808266
Epoch: 8032 Loss: 0.009239984676241875
Epoch: 8033 Loss: 0.009239024482667446
Epoch: 8034 Loss: 0.009237153455615044
Epoch: 8035 Loss: 0.0092362305149436
Epoch: 8036 Loss: 0.009233473800122738
Epoch: 8037 Loss: 0.009231405332684517
Epoch: 8038 Loss: 0.009229978546500206
Epoch: 8039 Loss: 0.009227588772773743
Epoch: 8040 Loss: 0.009226242080330849
Epoch: 8041 Loss: 0.00922463834285736
Epoch: 8042 Loss: 0.009223869070410728
Epoch: 8043 Loss: 0.009222528897225857
Epoch: 8044 Loss: 0.009218989871442318
Epoch: 8045 Loss: 0.009217959828674793
Epoch: 8046 Loss: 0.009215999394655228
Epoch: 8047 Loss: 0.009214497171342373
Epoch: 8048 Loss: 0.009211678057909012
Epoch: 8049 Loss: 0.009210501797497272
Epoch: 8050 Loss: 0.009208153933286667
Epoch: 8051 Loss: 0.009208625182509422
Epoch: 8052 Loss: 0.009205304086208344
Epoch: 8053 Loss: 0.009205318056046963
Epoch: 8054 Loss: 0.009202156215906143
Epoch: 8055 Loss: 0.009199688211083412
Epoch: 8056 Loss: 0.009199695661664009
Epoch: 8057 Loss: 0.009197215549647808
Epoch: 8058 Loss: 0.009195609018206596
Epoch: 8059 Loss: 0.009193435311317444
Epoch: 8060 Loss: 0.009191013872623444
Epoch: 8061 Loss: 0.009189970791339874
Epoch: 8062 Loss: 0.00918811745941639
Epoch: 8063 Loss: 0.00918575469404459
Epoch: 8064 Loss: 0.009184828959405422
Epoch: 8065 Loss: 0.009182436391711235
Epoch: 8066 Loss: 0.009181506931781769
Epoch: 8067 Loss: 0.009178990498185158
Epoch: 8068 Loss: 0.009177989326417446
Epoch: 8069 Loss: 0.009176651015877724
Epoch: 8070 Loss: 0.009174909442663193
Epoch: 8071 Loss: 0.009171957150101662
Epoch: 8072 Loss: 0.009170573204755783
Epoch: 8073 Loss: 0.009168921038508415
Epoch: 8074 Loss: 0.009167256765067577
Epoch: 8075 Loss: 0.009165425784885883
Epoch: 8076 Loss: 0.00916521716862917
Epoch: 8077 Loss: 0.009162853471934795
Epoch: 8078 Loss: 0.009159447625279427
Epoch: 8079 Loss: 0.009159654378890991
Epoch: 8080 Loss: 0.009156743995845318
Epoch: 8081 Loss: 0.00915481336414814
Epoch: 8082 Loss: 0.009152954444289207
Epoch: 8083 Loss: 0.009152091108262539
Epoch: 8084 Loss: 0.009151367470622063
Epoch: 8085 Loss: 0.009148815646767616
Epoch: 8086 Loss: 0.009147643111646175
Epoch: 8087 Loss: 0.009144612587988377
Epoch: 8088 Loss: 0.009142120368778706
Epoch: 8089 Loss: 0.009141623042523861
Epoch: 8090 Loss: 0.009139779955148697
Epoch: 8091 Loss: 0.009137923829257488
Epoch: 8092 Loss: 0.009136423468589783
Epoch: 8093 Loss: 0.009134862571954727
Epoch: 8094 Loss: 0.009132370352745056
Epoch: 8095 Loss: 0.00913019385188818
Epoch: 8096 Loss: 0.00912884995341301
Epoch: 8097 Loss: 0.009128241799771786
Epoch: 8098 Loss: 0.009126615710556507
Epoch: 8099 Loss: 0.0091239083558321
Epoch: 8100 Loss: 0.009121889248490334
Epoch: 8101 Loss: 0.009119889698922634
Epoch: 8102 Loss: 0.009119459427893162
Epoch: 8103 Loss: 0.009117873385548592
Epoch: 8104 Loss: 0.009115638211369514
Epoch: 8105 Loss: 0.00911357719451189
Epoch: 8106 Loss: 0.009112140163779259
Epoch: 8107 Loss: 0.009110639803111553
Epoch: 8108 Loss: 0.00910872407257557
Epoch: 8109 Loss: 0.009106592275202274
Epoch: 8110 Loss: 0.009105014614760876
Epoch: 8111 Loss: 0.009103469550609589
Epoch: 8112 Loss: 0.009102948009967804
Epoch: 8113 Loss: 0.009099029004573822
Epoch: 8114 Loss: 0.009097804315388203
Epoch: 8115 Loss: 0.009096226654946804
Epoch: 8116 Loss: 0.009095437824726105
Epoch: 8117 Loss: 0.009093879722058773
Epoch: 8118 Loss: 0.009092653170228004
Epoch: 8119 Loss: 0.009090718813240528
Epoch: 8120 Loss: 0.009088046848773956
Epoch: 8121 Loss: 0.00908685103058815
Epoch: 8122 Loss: 0.009084316901862621
Epoch: 8123 Loss: 0.009082717821002007
Epoch: 8124 Loss: 0.009081993252038956
Epoch: 8125 Loss: 0.009079830721020699
Epoch: 8126 Loss: 0.009076675400137901
Epoch: 8127 Loss: 0.00907600112259388
Epoch: 8128 Loss: 0.00907392892986536
Epoch: 8129 Loss: 0.009073128923773766
Epoch: 8130 Loss: 0.009072491899132729
Epoch: 8131 Loss: 0.009070081636309624
Epoch: 8132 Loss: 0.009066885337233543
Epoch: 8133 Loss: 0.009065801277756691
Epoch: 8134 Loss: 0.009064415469765663
Epoch: 8135 Loss: 0.009063373319804668
Epoch: 8136 Loss: 0.009061398915946484
Epoch: 8137 Loss: 0.009059080854058266
Epoch: 8138 Loss: 0.0090576047077775
Epoch: 8139 Loss: 0.009055644273757935
Epoch: 8140 Loss: 0.009054223075509071
Epoch: 8141 Loss: 0.009052000008523464
Epoch: 8142 Loss: 0.009049950167536736
Epoch: 8143 Loss: 0.009050878696143627
Epoch: 8144 Loss: 0.009047354571521282
Epoch: 8145 Loss: 0.009045126847922802
Epoch: 8146 Loss: 0.009043713100254536
Epoch: 8147 Loss: 0.009042289108037949
Epoch: 8148 Loss: 0.00904032588005066
Epoch: 8149 Loss: 0.009039823897182941
Epoch: 8150 Loss: 0.009036704897880554
Epoch: 8151 Loss: 0.009035308845341206
Epoch: 8152 Loss: 0.009034155867993832
Epoch: 8153 Loss: 0.009032382629811764
Epoch: 8154 Loss: 0.009031159803271294
Epoch: 8155 Loss: 0.009027877822518349
Epoch: 8156 Loss: 0.009026901796460152
Epoch: 8157 Loss: 0.009025141596794128
Epoch: 8158 Loss: 0.009023482911288738
Epoch: 8159 Loss: 0.009022027254104614
Epoch: 8160 Loss: 0.009020013734698296
Epoch: 8161 Loss: 0.009019250050187111
Epoch: 8162 Loss: 0.009016801603138447
Epoch: 8163 Loss: 0.009014904499053955
Epoch: 8164 Loss: 0.009012650698423386
Epoch: 8165 Loss: 0.009011508896946907
Epoch: 8166 Loss: 0.00900961086153984
Epoch: 8167 Loss: 0.009007793851196766
Epoch: 8168 Loss: 0.009005367755889893
Epoch: 8169 Loss: 0.009004621766507626
Epoch: 8170 Loss: 0.009003335610032082
Epoch: 8171 Loss: 0.009001326747238636
Epoch: 8172 Loss: 0.008999163284897804
Epoch: 8173 Loss: 0.008997365832328796
Epoch: 8174 Loss: 0.008996729739010334
Epoch: 8175 Loss: 0.008994599804282188
Epoch: 8176 Loss: 0.008993173018097878
Epoch: 8177 Loss: 0.008990779519081116
Epoch: 8178 Loss: 0.00898937787860632
Epoch: 8179 Loss: 0.008987460285425186
Epoch: 8180 Loss: 0.008986026979982853
Epoch: 8181 Loss: 0.008984656073153019
Epoch: 8182 Loss: 0.008981974795460701
Epoch: 8183 Loss: 0.008981618098914623
Epoch: 8184 Loss: 0.008978846482932568
Epoch: 8185 Loss: 0.008977279998362064
Epoch: 8186 Loss: 0.008975555188953876
Epoch: 8187 Loss: 0.008974776603281498
Epoch: 8188 Loss: 0.008971975184977055
Epoch: 8189 Loss: 0.008970977738499641
Epoch: 8190 Loss: 0.008969607762992382
Epoch: 8191 Loss: 0.008967174217104912
Epoch: 8192 Loss: 0.008964920416474342
Epoch: 8193 Loss: 0.008962406776845455
Epoch: 8194 Loss: 0.008961088955402374
Epoch: 8195 Loss: 0.00895951222628355
Epoch: 8196 Loss: 0.008957806043326855
Epoch: 8197 Loss: 0.00895531103014946
Epoch: 8198 Loss: 0.008953273296356201
Epoch: 8199 Loss: 0.008952438831329346
Epoch: 8200 Loss: 0.008950860239565372
Epoch: 8201 Loss: 0.008948031812906265
Epoch: 8202 Loss: 0.008947256952524185
Epoch: 8203 Loss: 0.008945169858634472
Epoch: 8204 Loss: 0.008942502550780773
Epoch: 8205 Loss: 0.008941036649048328
Epoch: 8206 Loss: 0.008939554914832115
Epoch: 8207 Loss: 0.008937172591686249
Epoch: 8208 Loss: 0.008935192599892616
Epoch: 8209 Loss: 0.008934194222092628
Epoch: 8210 Loss: 0.008931330405175686
Epoch: 8211 Loss: 0.008929564617574215
Epoch: 8212 Loss: 0.008928877301514149
Epoch: 8213 Loss: 0.008925551548600197
Epoch: 8214 Loss: 0.008924445137381554
Epoch: 8215 Loss: 0.008922514505684376
Epoch: 8216 Loss: 0.00892153475433588
Epoch: 8217 Loss: 0.00891896616667509
Epoch: 8218 Loss: 0.008916985243558884
Epoch: 8219 Loss: 0.008914715610444546
Epoch: 8220 Loss: 0.008913718163967133
Epoch: 8221 Loss: 0.008911428041756153
Epoch: 8222 Loss: 0.00891033187508583
Epoch: 8223 Loss: 0.008907443843781948
Epoch: 8224 Loss: 0.008907408453524113
Epoch: 8225 Loss: 0.008904979564249516
Epoch: 8226 Loss: 0.008903010748326778
Epoch: 8227 Loss: 0.008901397697627544
Epoch: 8228 Loss: 0.008899527601897717
Epoch: 8229 Loss: 0.0088970847427845
Epoch: 8230 Loss: 0.00889533944427967
Epoch: 8231 Loss: 0.008892880752682686
Epoch: 8232 Loss: 0.008891698904335499
Epoch: 8233 Loss: 0.008891310542821884
Epoch: 8234 Loss: 0.008888473734259605
Epoch: 8235 Loss: 0.00888720341026783
Epoch: 8236 Loss: 0.00888474564999342
Epoch: 8237 Loss: 0.008883013390004635
Epoch: 8238 Loss: 0.008881558664143085
Epoch: 8239 Loss: 0.008879324421286583
Epoch: 8240 Loss: 0.008877411484718323
Epoch: 8241 Loss: 0.008875174447894096
Epoch: 8242 Loss: 0.008873638696968555
Epoch: 8243 Loss: 0.008872626349329948
Epoch: 8244 Loss: 0.008870376273989677
Epoch: 8245 Loss: 0.008869928307831287
Epoch: 8246 Loss: 0.008866527117788792
Epoch: 8247 Loss: 0.00886448659002781
Epoch: 8248 Loss: 0.008863408118486404
Epoch: 8249 Loss: 0.0088624507188797
Epoch: 8250 Loss: 0.008860143832862377
Epoch: 8251 Loss: 0.008857307955622673
Epoch: 8252 Loss: 0.008856328204274178
Epoch: 8253 Loss: 0.008854780346155167
Epoch: 8254 Loss: 0.008852335624396801
Epoch: 8255 Loss: 0.008851748891174793
Epoch: 8256 Loss: 0.008850126527249813
Epoch: 8257 Loss: 0.008846614509820938
Epoch: 8258 Loss: 0.008845353499054909
Epoch: 8259 Loss: 0.00884492602199316
Epoch: 8260 Loss: 0.008841277100145817
Epoch: 8261 Loss: 0.008840560913085938
Epoch: 8262 Loss: 0.008838127367198467
Epoch: 8263 Loss: 0.008835709653794765
Epoch: 8264 Loss: 0.00883619487285614
Epoch: 8265 Loss: 0.008833050727844238
Epoch: 8266 Loss: 0.008830586448311806
Epoch: 8267 Loss: 0.008829211816191673
Epoch: 8268 Loss: 0.008827688172459602
Epoch: 8269 Loss: 0.008826776407659054
Epoch: 8270 Loss: 0.008823784068226814
Epoch: 8271 Loss: 0.008822333998978138
Epoch: 8272 Loss: 0.008821550756692886
Epoch: 8273 Loss: 0.008818666450679302
Epoch: 8274 Loss: 0.008817041292786598
Epoch: 8275 Loss: 0.008816120214760303
Epoch: 8276 Loss: 0.008814538829028606
Epoch: 8277 Loss: 0.00881152506917715
Epoch: 8278 Loss: 0.008811680600047112
Epoch: 8279 Loss: 0.008809036575257778
Epoch: 8280 Loss: 0.008807407692074776
Epoch: 8281 Loss: 0.008805046789348125
Epoch: 8282 Loss: 0.008802162483334541
Epoch: 8283 Loss: 0.00880197063088417
Epoch: 8284 Loss: 0.008800266310572624
Epoch: 8285 Loss: 0.008797809481620789
Epoch: 8286 Loss: 0.00879639945924282
Epoch: 8287 Loss: 0.008793668821454048
Epoch: 8288 Loss: 0.008792316541075706
Epoch: 8289 Loss: 0.008790943771600723
Epoch: 8290 Loss: 0.008789434097707272
Epoch: 8291 Loss: 0.008787882514297962
Epoch: 8292 Loss: 0.008786848746240139
Epoch: 8293 Loss: 0.008784228004515171
Epoch: 8294 Loss: 0.008782465010881424
Epoch: 8295 Loss: 0.0087810717523098
Epoch: 8296 Loss: 0.008780104108154774
Epoch: 8297 Loss: 0.008777300827205181
Epoch: 8298 Loss: 0.00877574272453785
Epoch: 8299 Loss: 0.008773870766162872
Epoch: 8300 Loss: 0.008771226741373539
Epoch: 8301 Loss: 0.008770850487053394
Epoch: 8302 Loss: 0.008767726831138134
Epoch: 8303 Loss: 0.008766211569309235
Epoch: 8304 Loss: 0.008765692822635174
Epoch: 8305 Loss: 0.008762757293879986
Epoch: 8306 Loss: 0.00876136589795351
Epoch: 8307 Loss: 0.008759560994803905
Epoch: 8308 Loss: 0.008758090436458588
Epoch: 8309 Loss: 0.00875735841691494
Epoch: 8310 Loss: 0.008754484355449677
Epoch: 8311 Loss: 0.008752620778977871
Epoch: 8312 Loss: 0.008751456625759602
Epoch: 8313 Loss: 0.008749950677156448
Epoch: 8314 Loss: 0.00874816533178091
Epoch: 8315 Loss: 0.00874563679099083
Epoch: 8316 Loss: 0.008744290098547935
Epoch: 8317 Loss: 0.008741925470530987
Epoch: 8318 Loss: 0.008741063065826893
Epoch: 8319 Loss: 0.008740110322833061
Epoch: 8320 Loss: 0.00873766839504242
Epoch: 8321 Loss: 0.008735871873795986
Epoch: 8322 Loss: 0.008735233917832375
Epoch: 8323 Loss: 0.008732393383979797
Epoch: 8324 Loss: 0.008730405010282993
Epoch: 8325 Loss: 0.008730094879865646
Epoch: 8326 Loss: 0.008728273212909698
Epoch: 8327 Loss: 0.008727092295885086
Epoch: 8328 Loss: 0.008724859915673733
Epoch: 8329 Loss: 0.00872267596423626
Epoch: 8330 Loss: 0.008721442893147469
Epoch: 8331 Loss: 0.00871930830180645
Epoch: 8332 Loss: 0.008717898279428482
Epoch: 8333 Loss: 0.00871653575450182
Epoch: 8334 Loss: 0.00871512945741415
Epoch: 8335 Loss: 0.008713195100426674
Epoch: 8336 Loss: 0.00871305726468563
Epoch: 8337 Loss: 0.008710702881217003
Epoch: 8338 Loss: 0.008708088658750057
Epoch: 8339 Loss: 0.008707661181688309
Epoch: 8340 Loss: 0.008706171996891499
Epoch: 8341 Loss: 0.008703379891812801
Epoch: 8342 Loss: 0.008702324703335762
Epoch: 8343 Loss: 0.008699972182512283
Epoch: 8344 Loss: 0.008699486963450909
Epoch: 8345 Loss: 0.008697666227817535
Epoch: 8346 Loss: 0.008695055730640888
Epoch: 8347 Loss: 0.008694554679095745
Epoch: 8348 Loss: 0.008692971430718899
Epoch: 8349 Loss: 0.008689749985933304
Epoch: 8350 Loss: 0.008689065463840961
Epoch: 8351 Loss: 0.008687240071594715
Epoch: 8352 Loss: 0.008685039356350899
Epoch: 8353 Loss: 0.008684031665325165
Epoch: 8354 Loss: 0.008682326413691044
Epoch: 8355 Loss: 0.008680864237248898
Epoch: 8356 Loss: 0.008680993691086769
Epoch: 8357 Loss: 0.008677639067173004
Epoch: 8358 Loss: 0.008675365708768368
Epoch: 8359 Loss: 0.008673671633005142
Epoch: 8360 Loss: 0.008672085590660572
Epoch: 8361 Loss: 0.008672618307173252
Epoch: 8362 Loss: 0.008669671602547169
Epoch: 8363 Loss: 0.008668317459523678
Epoch: 8364 Loss: 0.00866673793643713
Epoch: 8365 Loss: 0.008664768189191818
Epoch: 8366 Loss: 0.008663569577038288
Epoch: 8367 Loss: 0.008661465719342232
Epoch: 8368 Loss: 0.008660506457090378
Epoch: 8369 Loss: 0.00865907408297062
Epoch: 8370 Loss: 0.00865764357149601
Epoch: 8371 Loss: 0.008654577657580376
Epoch: 8372 Loss: 0.008653373457491398
Epoch: 8373 Loss: 0.008652682416141033
Epoch: 8374 Loss: 0.008650707080960274
Epoch: 8375 Loss: 0.008649223484098911
Epoch: 8376 Loss: 0.008647475391626358
Epoch: 8377 Loss: 0.008645443245768547
Epoch: 8378 Loss: 0.008643444627523422
Epoch: 8379 Loss: 0.008642296306788921
Epoch: 8380 Loss: 0.008640066720545292
Epoch: 8381 Loss: 0.008639286272227764
Epoch: 8382 Loss: 0.008638344705104828
Epoch: 8383 Loss: 0.00863660965114832
Epoch: 8384 Loss: 0.008633731864392757
Epoch: 8385 Loss: 0.008631773293018341
Epoch: 8386 Loss: 0.008631652221083641
Epoch: 8387 Loss: 0.008630011230707169
Epoch: 8388 Loss: 0.008627403527498245
Epoch: 8389 Loss: 0.008626366965472698
Epoch: 8390 Loss: 0.00862548965960741
Epoch: 8391 Loss: 0.008624170906841755
Epoch: 8392 Loss: 0.008621245622634888
Epoch: 8393 Loss: 0.00862060021609068
Epoch: 8394 Loss: 0.008618335239589214
Epoch: 8395 Loss: 0.008617071434855461
Epoch: 8396 Loss: 0.008615237660706043
Epoch: 8397 Loss: 0.00861372146755457
Epoch: 8398 Loss: 0.008611277677118778
Epoch: 8399 Loss: 0.008611101657152176
Epoch: 8400 Loss: 0.008609514683485031
Epoch: 8401 Loss: 0.008606714196503162
Epoch: 8402 Loss: 0.008605712093412876
Epoch: 8403 Loss: 0.00860399566590786
Epoch: 8404 Loss: 0.008601714856922626
Epoch: 8405 Loss: 0.0086015984416008
Epoch: 8406 Loss: 0.00859977025538683
Epoch: 8407 Loss: 0.008597822859883308
Epoch: 8408 Loss: 0.008596590720117092
Epoch: 8409 Loss: 0.008595092222094536
Epoch: 8410 Loss: 0.00859417486935854
Epoch: 8411 Loss: 0.00859171710908413
Epoch: 8412 Loss: 0.008589111268520355
Epoch: 8413 Loss: 0.008589022792875767
Epoch: 8414 Loss: 0.008587137795984745
Epoch: 8415 Loss: 0.008584675379097462
Epoch: 8416 Loss: 0.008583293296396732
Epoch: 8417 Loss: 0.00858138408511877
Epoch: 8418 Loss: 0.008581256493926048
Epoch: 8419 Loss: 0.00857924297451973
Epoch: 8420 Loss: 0.00857679732143879
Epoch: 8421 Loss: 0.008576675318181515
Epoch: 8422 Loss: 0.008574494160711765
Epoch: 8423 Loss: 0.008571848273277283
Epoch: 8424 Loss: 0.008571330457925797
Epoch: 8425 Loss: 0.008570518344640732
Epoch: 8426 Loss: 0.008568507619202137
Epoch: 8427 Loss: 0.008566760458052158
Epoch: 8428 Loss: 0.008564666844904423
Epoch: 8429 Loss: 0.008562888950109482
Epoch: 8430 Loss: 0.008562164381146431
Epoch: 8431 Loss: 0.008560623973608017
Epoch: 8432 Loss: 0.008558041416108608
Epoch: 8433 Loss: 0.008557087741792202
Epoch: 8434 Loss: 0.008555863983929157
Epoch: 8435 Loss: 0.008554457686841488
Epoch: 8436 Loss: 0.00855227094143629
Epoch: 8437 Loss: 0.00855129212141037
Epoch: 8438 Loss: 0.008550194092094898
Epoch: 8439 Loss: 0.00854698196053505
Epoch: 8440 Loss: 0.008546733297407627
Epoch: 8441 Loss: 0.008544361218810081
Epoch: 8442 Loss: 0.0085433479398489
Epoch: 8443 Loss: 0.008540538139641285
Epoch: 8444 Loss: 0.008539346978068352
Epoch: 8445 Loss: 0.008538124151527882
Epoch: 8446 Loss: 0.008537490852177143
Epoch: 8447 Loss: 0.008535915054380894
Epoch: 8448 Loss: 0.00853488128632307
Epoch: 8449 Loss: 0.008532365784049034
Epoch: 8450 Loss: 0.008530611172318459
Epoch: 8451 Loss: 0.008528521284461021
Epoch: 8452 Loss: 0.00852725375443697
Epoch: 8453 Loss: 0.008525444194674492
Epoch: 8454 Loss: 0.008524088189005852
Epoch: 8455 Loss: 0.008523484691977501
Epoch: 8456 Loss: 0.00852120015770197
Epoch: 8457 Loss: 0.008520129136741161
Epoch: 8458 Loss: 0.008517618291079998
Epoch: 8459 Loss: 0.008516980335116386
Epoch: 8460 Loss: 0.00851490069180727
Epoch: 8461 Loss: 0.008514591492712498
Epoch: 8462 Loss: 0.008512658067047596
Epoch: 8463 Loss: 0.008511075749993324
Epoch: 8464 Loss: 0.0085093192756176
Epoch: 8465 Loss: 0.00850763637572527
Epoch: 8466 Loss: 0.008505884557962418
Epoch: 8467 Loss: 0.008504237048327923
Epoch: 8468 Loss: 0.008502057753503323
Epoch: 8469 Loss: 0.008501236326992512
Epoch: 8470 Loss: 0.008499855175614357
Epoch: 8471 Loss: 0.008498125709593296
Epoch: 8472 Loss: 0.008496544323861599
Epoch: 8473 Loss: 0.008494781330227852
Epoch: 8474 Loss: 0.008493132889270782
Epoch: 8475 Loss: 0.008493098430335522
Epoch: 8476 Loss: 0.00849011167883873
Epoch: 8477 Loss: 0.008488263003528118
Epoch: 8478 Loss: 0.00848735123872757
Epoch: 8479 Loss: 0.008486149832606316
Epoch: 8480 Loss: 0.008484553545713425
Epoch: 8481 Loss: 0.008483542129397392
Epoch: 8482 Loss: 0.008481422439217567
Epoch: 8483 Loss: 0.008479434065520763
Epoch: 8484 Loss: 0.00847847480326891
Epoch: 8485 Loss: 0.0084762554615736
Epoch: 8486 Loss: 0.008475849404931068
Epoch: 8487 Loss: 0.008474687114357948
Epoch: 8488 Loss: 0.008472301065921783
Epoch: 8489 Loss: 0.008469972759485245
Epoch: 8490 Loss: 0.008468922227621078
Epoch: 8491 Loss: 0.008467529900372028
Epoch: 8492 Loss: 0.008466127328574657
Epoch: 8493 Loss: 0.008465352468192577
Epoch: 8494 Loss: 0.008462296798825264
Epoch: 8495 Loss: 0.00846134964376688
Epoch: 8496 Loss: 0.00846056267619133
Epoch: 8497 Loss: 0.008458597585558891
Epoch: 8498 Loss: 0.008456907235085964
Epoch: 8499 Loss: 0.008455224335193634
Epoch: 8500 Loss: 0.00845399685204029
Epoch: 8501 Loss: 0.008451990783214569
Epoch: 8502 Loss: 0.008450099267065525
Epoch: 8503 Loss: 0.008449790999293327
Epoch: 8504 Loss: 0.008448067121207714
Epoch: 8505 Loss: 0.008446351625025272
Epoch: 8506 Loss: 0.008443465456366539
Epoch: 8507 Loss: 0.008442860096693039
Epoch: 8508 Loss: 0.008442659862339497
Epoch: 8509 Loss: 0.008440050296485424
Epoch: 8510 Loss: 0.00843902863562107
Epoch: 8511 Loss: 0.008437828160822392
Epoch: 8512 Loss: 0.008435884490609169
Epoch: 8513 Loss: 0.00843342486768961
Epoch: 8514 Loss: 0.008432616479694843
Epoch: 8515 Loss: 0.008430184796452522
Epoch: 8516 Loss: 0.008429468609392643
Epoch: 8517 Loss: 0.008427688851952553
Epoch: 8518 Loss: 0.008426206186413765
Epoch: 8519 Loss: 0.008425043895840645
Epoch: 8520 Loss: 0.008423338644206524
Epoch: 8521 Loss: 0.008421314880251884
Epoch: 8522 Loss: 0.008419979363679886
Epoch: 8523 Loss: 0.008419865742325783
Epoch: 8524 Loss: 0.00841678585857153
Epoch: 8525 Loss: 0.008414935320615768
Epoch: 8526 Loss: 0.008414666168391705
Epoch: 8527 Loss: 0.008413179777562618
Epoch: 8528 Loss: 0.008410830982029438
Epoch: 8529 Loss: 0.00841051246970892
Epoch: 8530 Loss: 0.008408376015722752
Epoch: 8531 Loss: 0.008406744338572025
Epoch: 8532 Loss: 0.008405881933867931
Epoch: 8533 Loss: 0.008403649553656578
Epoch: 8534 Loss: 0.008402249775826931
Epoch: 8535 Loss: 0.00839968491345644
Epoch: 8536 Loss: 0.00839932169765234
Epoch: 8537 Loss: 0.008397288620471954
Epoch: 8538 Loss: 0.00839566346257925
Epoch: 8539 Loss: 0.008395004086196423
Epoch: 8540 Loss: 0.008393767289817333
Epoch: 8541 Loss: 0.008391840383410454
Epoch: 8542 Loss: 0.008390520699322224
Epoch: 8543 Loss: 0.008388219401240349
Epoch: 8544 Loss: 0.008387749083340168
Epoch: 8545 Loss: 0.008384895510971546
Epoch: 8546 Loss: 0.008383956737816334
Epoch: 8547 Loss: 0.008382311090826988
Epoch: 8548 Loss: 0.008381997235119343
Epoch: 8549 Loss: 0.008380421437323093
Epoch: 8550 Loss: 0.008378553204238415
Epoch: 8551 Loss: 0.00837642326951027
Epoch: 8552 Loss: 0.008375299163162708
Epoch: 8553 Loss: 0.008372889831662178
Epoch: 8554 Loss: 0.008371453732252121
Epoch: 8555 Loss: 0.00837196409702301
Epoch: 8556 Loss: 0.008369147777557373
Epoch: 8557 Loss: 0.00836815033107996
Epoch: 8558 Loss: 0.0083659952506423
Epoch: 8559 Loss: 0.008364464156329632
Epoch: 8560 Loss: 0.00836339220404625
Epoch: 8561 Loss: 0.008361984975636005
Epoch: 8562 Loss: 0.008359360508620739
Epoch: 8563 Loss: 0.008358272723853588
Epoch: 8564 Loss: 0.008358118124306202
Epoch: 8565 Loss: 0.008356439881026745
Epoch: 8566 Loss: 0.008354713208973408
Epoch: 8567 Loss: 0.00835246592760086
Epoch: 8568 Loss: 0.00835160817950964
Epoch: 8569 Loss: 0.008349433541297913
Epoch: 8570 Loss: 0.008347425609827042
Epoch: 8571 Loss: 0.00834619626402855
Epoch: 8572 Loss: 0.00834617204964161
Epoch: 8573 Loss: 0.008343965746462345
Epoch: 8574 Loss: 0.00834288727492094
Epoch: 8575 Loss: 0.008340885862708092
Epoch: 8576 Loss: 0.008338685147464275
Epoch: 8577 Loss: 0.008337373845279217
Epoch: 8578 Loss: 0.008335969410836697
Epoch: 8579 Loss: 0.008334926329553127
Epoch: 8580 Loss: 0.008333074860274792
Epoch: 8581 Loss: 0.008332227356731892
Epoch: 8582 Loss: 0.008330417796969414
Epoch: 8583 Loss: 0.008328069001436234
Epoch: 8584 Loss: 0.008327288553118706
Epoch: 8585 Loss: 0.00832608062773943
Epoch: 8586 Loss: 0.008324924856424332
Epoch: 8587 Loss: 0.008323606103658676
Epoch: 8588 Loss: 0.008321329951286316
Epoch: 8589 Loss: 0.008319737389683723
Epoch: 8590 Loss: 0.008317917585372925
Epoch: 8591 Loss: 0.00831591710448265
Epoch: 8592 Loss: 0.00831575132906437
Epoch: 8593 Loss: 0.008314032107591629
Epoch: 8594 Loss: 0.008313886821269989
Epoch: 8595 Loss: 0.008310841396450996
Epoch: 8596 Loss: 0.008309971541166306
Epoch: 8597 Loss: 0.008307671174407005
Epoch: 8598 Loss: 0.008306819014251232
Epoch: 8599 Loss: 0.008305069990456104
Epoch: 8600 Loss: 0.008303521201014519
Epoch: 8601 Loss: 0.008303006179630756
Epoch: 8602 Loss: 0.008301538415253162
Epoch: 8603 Loss: 0.008299075998365879
Epoch: 8604 Loss: 0.008297417312860489
Epoch: 8605 Loss: 0.008296252228319645
Epoch: 8606 Loss: 0.008295436389744282
Epoch: 8607 Loss: 0.00829394068568945
Epoch: 8608 Loss: 0.008291580714285374
Epoch: 8609 Loss: 0.008289577439427376
Epoch: 8610 Loss: 0.008288927376270294
Epoch: 8611 Loss: 0.008286682888865471
Epoch: 8612 Loss: 0.008285661228001118
Epoch: 8613 Loss: 0.008284968324005604
Epoch: 8614 Loss: 0.008282912895083427
Epoch: 8615 Loss: 0.00828131940215826
Epoch: 8616 Loss: 0.00828026607632637
Epoch: 8617 Loss: 0.008279308676719666
Epoch: 8618 Loss: 0.008277376182377338
Epoch: 8619 Loss: 0.008275743573904037
Epoch: 8620 Loss: 0.008273906074464321
Epoch: 8621 Loss: 0.008272486738860607
Epoch: 8622 Loss: 0.008271357044577599
Epoch: 8623 Loss: 0.008269339799880981
Epoch: 8624 Loss: 0.008269436657428741
Epoch: 8625 Loss: 0.008266723714768887
Epoch: 8626 Loss: 0.008265930227935314
Epoch: 8627 Loss: 0.008264689706265926
Epoch: 8628 Loss: 0.008263244293630123
Epoch: 8629 Loss: 0.008261773735284805
Epoch: 8630 Loss: 0.008259306661784649
Epoch: 8631 Loss: 0.0082578519359231
Epoch: 8632 Loss: 0.008256923407316208
Epoch: 8633 Loss: 0.008254408836364746
Epoch: 8634 Loss: 0.008253048174083233
Epoch: 8635 Loss: 0.008252415806055069
Epoch: 8636 Loss: 0.008251185528934002
Epoch: 8637 Loss: 0.008249837905168533
Epoch: 8638 Loss: 0.008248438127338886
Epoch: 8639 Loss: 0.008245616219937801
Epoch: 8640 Loss: 0.008244755677878857
Epoch: 8641 Loss: 0.008243156597018242
Epoch: 8642 Loss: 0.008242294192314148
Epoch: 8643 Loss: 0.008240777999162674
Epoch: 8644 Loss: 0.008239539340138435
Epoch: 8645 Loss: 0.008237103000283241
Epoch: 8646 Loss: 0.008235792629420757
Epoch: 8647 Loss: 0.008234669454395771
Epoch: 8648 Loss: 0.008233691565692425
Epoch: 8649 Loss: 0.008232277818024158
Epoch: 8650 Loss: 0.008231374435126781
Epoch: 8651 Loss: 0.008228855207562447
Epoch: 8652 Loss: 0.008228300139307976
Epoch: 8653 Loss: 0.008226171135902405
Epoch: 8654 Loss: 0.008224689401686192
Epoch: 8655 Loss: 0.008223186247050762
Epoch: 8656 Loss: 0.008222188800573349
Epoch: 8657 Loss: 0.008219773881137371
Epoch: 8658 Loss: 0.008218802511692047
Epoch: 8659 Loss: 0.008217073045670986
Epoch: 8660 Loss: 0.008215853944420815
Epoch: 8661 Loss: 0.00821480993181467
Epoch: 8662 Loss: 0.008212614804506302
Epoch: 8663 Loss: 0.008211175911128521
Epoch: 8664 Loss: 0.008209578692913055
Epoch: 8665 Loss: 0.008208652026951313
Epoch: 8666 Loss: 0.008208026178181171
Epoch: 8667 Loss: 0.00820594560354948
Epoch: 8668 Loss: 0.00820403452962637
Epoch: 8669 Loss: 0.008203266188502312
Epoch: 8670 Loss: 0.008201759308576584
Epoch: 8671 Loss: 0.00819986592978239
Epoch: 8672 Loss: 0.008197612129151821
Epoch: 8673 Loss: 0.008196678943932056
Epoch: 8674 Loss: 0.00819474644958973
Epoch: 8675 Loss: 0.008194085210561752
Epoch: 8676 Loss: 0.008194255642592907
Epoch: 8677 Loss: 0.008191447705030441
Epoch: 8678 Loss: 0.0081891855224967
Epoch: 8679 Loss: 0.008188367821276188
Epoch: 8680 Loss: 0.008188210427761078
Epoch: 8681 Loss: 0.008185976184904575
Epoch: 8682 Loss: 0.008183260448276997
Epoch: 8683 Loss: 0.008182685822248459
Epoch: 8684 Loss: 0.008181611075997353
Epoch: 8685 Loss: 0.008180397562682629
Epoch: 8686 Loss: 0.008178405463695526
Epoch: 8687 Loss: 0.008176975883543491
Epoch: 8688 Loss: 0.008175867609679699
Epoch: 8689 Loss: 0.008173881098628044
Epoch: 8690 Loss: 0.008172152563929558
Epoch: 8691 Loss: 0.00817091390490532
Epoch: 8692 Loss: 0.008169647306203842
Epoch: 8693 Loss: 0.008168188855051994
Epoch: 8694 Loss: 0.008166752755641937
Epoch: 8695 Loss: 0.008165429346263409
Epoch: 8696 Loss: 0.008165271952748299
Epoch: 8697 Loss: 0.008161916397511959
Epoch: 8698 Loss: 0.008161721751093864
Epoch: 8699 Loss: 0.008160091936588287
Epoch: 8700 Loss: 0.008157609961926937
Epoch: 8701 Loss: 0.008157881908118725
Epoch: 8702 Loss: 0.008154899813234806
Epoch: 8703 Loss: 0.008153854869306087
Epoch: 8704 Loss: 0.008152149617671967
Epoch: 8705 Loss: 0.008151769638061523
Epoch: 8706 Loss: 0.008149934932589531
Epoch: 8707 Loss: 0.0081482520326972
Epoch: 8708 Loss: 0.00814602430909872
Epoch: 8709 Loss: 0.008145459927618504
Epoch: 8710 Loss: 0.008143384009599686
Epoch: 8711 Loss: 0.008143831975758076
Epoch: 8712 Loss: 0.008141846396028996
Epoch: 8713 Loss: 0.008140658028423786
Epoch: 8714 Loss: 0.008138205856084824
Epoch: 8715 Loss: 0.008137053810060024
Epoch: 8716 Loss: 0.008135206066071987
Epoch: 8717 Loss: 0.00813376996666193
Epoch: 8718 Loss: 0.008132735267281532
Epoch: 8719 Loss: 0.008132308721542358
Epoch: 8720 Loss: 0.008129649795591831
Epoch: 8721 Loss: 0.008127691224217415
Epoch: 8722 Loss: 0.008126474916934967
Epoch: 8723 Loss: 0.008125306107103825
Epoch: 8724 Loss: 0.008124385960400105
Epoch: 8725 Loss: 0.0081223975867033
Epoch: 8726 Loss: 0.008121823891997337
Epoch: 8727 Loss: 0.008120167069137096
Epoch: 8728 Loss: 0.00811837799847126
Epoch: 8729 Loss: 0.00811735074967146
Epoch: 8730 Loss: 0.008115519769489765
Epoch: 8731 Loss: 0.00811397098004818
Epoch: 8732 Loss: 0.00811233650892973
Epoch: 8733 Loss: 0.008111194707453251
Epoch: 8734 Loss: 0.008109855465590954
Epoch: 8735 Loss: 0.008109590038657188
Epoch: 8736 Loss: 0.008107581175863743
Epoch: 8737 Loss: 0.008106614463031292
Epoch: 8738 Loss: 0.008105171844363213
Epoch: 8739 Loss: 0.008103444240987301
Epoch: 8740 Loss: 0.008100549690425396
Epoch: 8741 Loss: 0.008100100792944431
Epoch: 8742 Loss: 0.00809833500534296
Epoch: 8743 Loss: 0.008098059333860874
Epoch: 8744 Loss: 0.0080957543104887
Epoch: 8745 Loss: 0.008094166405498981
Epoch: 8746 Loss: 0.008093985728919506
Epoch: 8747 Loss: 0.008092071861028671
Epoch: 8748 Loss: 0.008090605959296227
Epoch: 8749 Loss: 0.008088797330856323
Epoch: 8750 Loss: 0.00808660127222538
Epoch: 8751 Loss: 0.00808638520538807
Epoch: 8752 Loss: 0.008084657602012157
Epoch: 8753 Loss: 0.00808244850486517
Epoch: 8754 Loss: 0.008082631975412369
Epoch: 8755 Loss: 0.008079798892140388
Epoch: 8756 Loss: 0.008078821934759617
Epoch: 8757 Loss: 0.008077169768512249
Epoch: 8758 Loss: 0.0080768633633852
Epoch: 8759 Loss: 0.008074726909399033
Epoch: 8760 Loss: 0.008073524571955204
Epoch: 8761 Loss: 0.008072203025221825
Epoch: 8762 Loss: 0.008070466108620167
Epoch: 8763 Loss: 0.00806967169046402
Epoch: 8764 Loss: 0.008067797869443893
Epoch: 8765 Loss: 0.008066225796937943
Epoch: 8766 Loss: 0.008064810186624527
Epoch: 8767 Loss: 0.008065075613558292
Epoch: 8768 Loss: 0.00806193333119154
Epoch: 8769 Loss: 0.008060915395617485
Epoch: 8770 Loss: 0.008059474639594555
Epoch: 8771 Loss: 0.008058147504925728
Epoch: 8772 Loss: 0.008056889288127422
Epoch: 8773 Loss: 0.008055462501943111
Epoch: 8774 Loss: 0.008054361678659916
Epoch: 8775 Loss: 0.008051942102611065
Epoch: 8776 Loss: 0.008050587028265
Epoch: 8777 Loss: 0.00805017352104187
Epoch: 8778 Loss: 0.008048665709793568
Epoch: 8779 Loss: 0.008046984672546387
Epoch: 8780 Loss: 0.008045864291489124
Epoch: 8781 Loss: 0.008043952286243439
Epoch: 8782 Loss: 0.008043264970183372
Epoch: 8783 Loss: 0.008041221648454666
Epoch: 8784 Loss: 0.008039874956011772
Epoch: 8785 Loss: 0.008038503117859364
Epoch: 8786 Loss: 0.008036664687097073
Epoch: 8787 Loss: 0.008035793900489807
Epoch: 8788 Loss: 0.00803512241691351
Epoch: 8789 Loss: 0.008033308200538158
Epoch: 8790 Loss: 0.008031546138226986
Epoch: 8791 Loss: 0.008029932156205177
Epoch: 8792 Loss: 0.008028443902730942
Epoch: 8793 Loss: 0.008027208037674427
Epoch: 8794 Loss: 0.008025508373975754
Epoch: 8795 Loss: 0.008025847375392914
Epoch: 8796 Loss: 0.008022664114832878
Epoch: 8797 Loss: 0.008021984249353409
Epoch: 8798 Loss: 0.008020332083106041
Epoch: 8799 Loss: 0.008020436391234398
Epoch: 8800 Loss: 0.008018219843506813
Epoch: 8801 Loss: 0.00801668781787157
Epoch: 8802 Loss: 0.008015102706849575
Epoch: 8803 Loss: 0.008013799786567688
Epoch: 8804 Loss: 0.008012480102479458
Epoch: 8805 Loss: 0.008011296391487122
Epoch: 8806 Loss: 0.008010243065655231
Epoch: 8807 Loss: 0.008007516153156757
Epoch: 8808 Loss: 0.008006684482097626
Epoch: 8809 Loss: 0.008005748502910137
Epoch: 8810 Loss: 0.008004066534340382
Epoch: 8811 Loss: 0.008002689108252525
Epoch: 8812 Loss: 0.008001377806067467
Epoch: 8813 Loss: 0.008000602945685387
Epoch: 8814 Loss: 0.00799831748008728
Epoch: 8815 Loss: 0.007996520027518272
Epoch: 8816 Loss: 0.007995062507689
Epoch: 8817 Loss: 0.007993904873728752
Epoch: 8818 Loss: 0.00799303688108921
Epoch: 8819 Loss: 0.007991580292582512
Epoch: 8820 Loss: 0.007990963757038116
Epoch: 8821 Loss: 0.007988856174051762
Epoch: 8822 Loss: 0.007987621240317822
Epoch: 8823 Loss: 0.007986277341842651
Epoch: 8824 Loss: 0.007983648218214512
Epoch: 8825 Loss: 0.007983889430761337
Epoch: 8826 Loss: 0.007981614209711552
Epoch: 8827 Loss: 0.00798027217388153
Epoch: 8828 Loss: 0.007980183698236942
Epoch: 8829 Loss: 0.007978212088346481
Epoch: 8830 Loss: 0.00797695480287075
Epoch: 8831 Loss: 0.007974502630531788
Epoch: 8832 Loss: 0.007974669337272644
Epoch: 8833 Loss: 0.007971834391355515
Epoch: 8834 Loss: 0.007971296086907387
Epoch: 8835 Loss: 0.007969742640852928
Epoch: 8836 Loss: 0.007968406192958355
Epoch: 8837 Loss: 0.007967215031385422
Epoch: 8838 Loss: 0.007965365424752235
Epoch: 8839 Loss: 0.007963444106280804
Epoch: 8840 Loss: 0.007962577976286411
Epoch: 8841 Loss: 0.007960823364555836
Epoch: 8842 Loss: 0.00795956514775753
Epoch: 8843 Loss: 0.007959303446114063
Epoch: 8844 Loss: 0.007958250120282173
Epoch: 8845 Loss: 0.007955826818943024
Epoch: 8846 Loss: 0.007953489199280739
Epoch: 8847 Loss: 0.00795278511941433
Epoch: 8848 Loss: 0.007951583713293076
Epoch: 8849 Loss: 0.007950272411108017
Epoch: 8850 Loss: 0.007948272861540318
Epoch: 8851 Loss: 0.007948307320475578
Epoch: 8852 Loss: 0.007945445366203785
Epoch: 8853 Loss: 0.0079454081133008
Epoch: 8854 Loss: 0.007942209020256996
Epoch: 8855 Loss: 0.007941251620650291
Epoch: 8856 Loss: 0.007939699105918407
Epoch: 8857 Loss: 0.007939102128148079
Epoch: 8858 Loss: 0.007937472313642502
Epoch: 8859 Loss: 0.00793667882680893
Epoch: 8860 Loss: 0.007934915833175182
Epoch: 8861 Loss: 0.00793179590255022
Epoch: 8862 Loss: 0.007931658998131752
Epoch: 8863 Loss: 0.007930127903819084
Epoch: 8864 Loss: 0.007929444313049316
Epoch: 8865 Loss: 0.007928062230348587
Epoch: 8866 Loss: 0.00792558304965496
Epoch: 8867 Loss: 0.007924450561404228
Epoch: 8868 Loss: 0.007923043332993984
Epoch: 8869 Loss: 0.00792175717651844
Epoch: 8870 Loss: 0.007920386269688606
Epoch: 8871 Loss: 0.007919189520180225
Epoch: 8872 Loss: 0.00791760440915823
Epoch: 8873 Loss: 0.007916150614619255
Epoch: 8874 Loss: 0.007914168760180473
Epoch: 8875 Loss: 0.007913612760603428
Epoch: 8876 Loss: 0.007911844179034233
Epoch: 8877 Loss: 0.007910037413239479
Epoch: 8878 Loss: 0.00790936779230833
Epoch: 8879 Loss: 0.007907447405159473
Epoch: 8880 Loss: 0.007906627841293812
Epoch: 8881 Loss: 0.007904743775725365
Epoch: 8882 Loss: 0.00790483970195055
Epoch: 8883 Loss: 0.00790167972445488
Epoch: 8884 Loss: 0.007900475524365902
Epoch: 8885 Loss: 0.007899755612015724
Epoch: 8886 Loss: 0.007899140007793903
Epoch: 8887 Loss: 0.007897616364061832
Epoch: 8888 Loss: 0.0078953942283988
Epoch: 8889 Loss: 0.007893840782344341
Epoch: 8890 Loss: 0.007892604917287827
Epoch: 8891 Loss: 0.007891124114394188
Epoch: 8892 Loss: 0.007890515960752964
Epoch: 8893 Loss: 0.007888434454798698
Epoch: 8894 Loss: 0.007886146195232868
Epoch: 8895 Loss: 0.007886648178100586
Epoch: 8896 Loss: 0.00788432639092207
Epoch: 8897 Loss: 0.007882799953222275
Epoch: 8898 Loss: 0.00788146536797285
Epoch: 8899 Loss: 0.007880917750298977
Epoch: 8900 Loss: 0.007878211326897144
Epoch: 8901 Loss: 0.007877863943576813
Epoch: 8902 Loss: 0.00787578895688057
Epoch: 8903 Loss: 0.00787446741014719
Epoch: 8904 Loss: 0.007873971946537495
Epoch: 8905 Loss: 0.007873175665736198
Epoch: 8906 Loss: 0.007870767265558243
Epoch: 8907 Loss: 0.00786868017166853
Epoch: 8908 Loss: 0.007867211475968361
Epoch: 8909 Loss: 0.00786615815013647
Epoch: 8910 Loss: 0.007864943705499172
Epoch: 8911 Loss: 0.00786399096250534
Epoch: 8912 Loss: 0.007862267084419727
Epoch: 8913 Loss: 0.007861025631427765
Epoch: 8914 Loss: 0.007859895005822182
Epoch: 8915 Loss: 0.007858287543058395
Epoch: 8916 Loss: 0.007856673561036587
Epoch: 8917 Loss: 0.007855753414332867
Epoch: 8918 Loss: 0.007853608578443527
Epoch: 8919 Loss: 0.007852145470678806
Epoch: 8920 Loss: 0.007852236740291119
Epoch: 8921 Loss: 0.007850228808820248
Epoch: 8922 Loss: 0.0078481025993824
Epoch: 8923 Loss: 0.007847681641578674
Epoch: 8924 Loss: 0.007845663465559483
Epoch: 8925 Loss: 0.007844044826924801
Epoch: 8926 Loss: 0.007842376828193665
Epoch: 8927 Loss: 0.00784269254654646
Epoch: 8928 Loss: 0.007840950042009354
Epoch: 8929 Loss: 0.007838979363441467
Epoch: 8930 Loss: 0.00783715769648552
Epoch: 8931 Loss: 0.007836131379008293
Epoch: 8932 Loss: 0.007835003547370434
Epoch: 8933 Loss: 0.007832763716578484
Epoch: 8934 Loss: 0.007832051254808903
Epoch: 8935 Loss: 0.007830983959138393
Epoch: 8936 Loss: 0.007830211892724037
Epoch: 8937 Loss: 0.007827792316675186
Epoch: 8938 Loss: 0.007826345041394234
Epoch: 8939 Loss: 0.007825986482203007
Epoch: 8940 Loss: 0.007824615575373173
Epoch: 8941 Loss: 0.00782299879938364
Epoch: 8942 Loss: 0.007821250706911087
Epoch: 8943 Loss: 0.007819334045052528
Epoch: 8944 Loss: 0.007818488404154778
Epoch: 8945 Loss: 0.007816418074071407
Epoch: 8946 Loss: 0.007815981283783913
Epoch: 8947 Loss: 0.007815190590918064
Epoch: 8948 Loss: 0.007812971249222755
Epoch: 8949 Loss: 0.007812232710421085
Epoch: 8950 Loss: 0.007810893002897501
Epoch: 8951 Loss: 0.007808605674654245
Epoch: 8952 Loss: 0.007807884365320206
Epoch: 8953 Loss: 0.007806005422025919
Epoch: 8954 Loss: 0.007805346045643091
Epoch: 8955 Loss: 0.007803051266819239
Epoch: 8956 Loss: 0.007802503649145365
Epoch: 8957 Loss: 0.007800820283591747
Epoch: 8958 Loss: 0.007799766957759857
Epoch: 8959 Loss: 0.007798576261848211
Epoch: 8960 Loss: 0.00779760954901576
Epoch: 8961 Loss: 0.007796702906489372
Epoch: 8962 Loss: 0.007794040720909834
Epoch: 8963 Loss: 0.007792463060468435
Epoch: 8964 Loss: 0.00779183441773057
Epoch: 8965 Loss: 0.007789767347276211
Epoch: 8966 Loss: 0.007788920775055885
Epoch: 8967 Loss: 0.007787465583533049
Epoch: 8968 Loss: 0.007785510737448931
Epoch: 8969 Loss: 0.007785164751112461
Epoch: 8970 Loss: 0.0077848779037594795
Epoch: 8971 Loss: 0.00778209650889039
Epoch: 8972 Loss: 0.00778167461976409
Epoch: 8973 Loss: 0.007779930718243122
Epoch: 8974 Loss: 0.007778809871524572
Epoch: 8975 Loss: 0.007776495069265366
Epoch: 8976 Loss: 0.007775250822305679
Epoch: 8977 Loss: 0.007773739751428366
Epoch: 8978 Loss: 0.007773412857204676
Epoch: 8979 Loss: 0.007770810276269913
Epoch: 8980 Loss: 0.007770364638417959
Epoch: 8981 Loss: 0.007768956013023853
Epoch: 8982 Loss: 0.007767498027533293
Epoch: 8983 Loss: 0.007766513153910637
Epoch: 8984 Loss: 0.0077650658786296844
Epoch: 8985 Loss: 0.007764259818941355
Epoch: 8986 Loss: 0.007761732209473848
Epoch: 8987 Loss: 0.007760156877338886
Epoch: 8988 Loss: 0.0077596246264874935
Epoch: 8989 Loss: 0.00775811867788434
Epoch: 8990 Loss: 0.0077570402063429356
Epoch: 8991 Loss: 0.00775576289743185
Epoch: 8992 Loss: 0.007754711899906397
Epoch: 8993 Loss: 0.007753328420221806
Epoch: 8994 Loss: 0.007752023637294769
Epoch: 8995 Loss: 0.007749089505523443
Epoch: 8996 Loss: 0.007748234085738659
Epoch: 8997 Loss: 0.007747931871563196
Epoch: 8998 Loss: 0.007745574228465557
Epoch: 8999 Loss: 0.007744916714727879
Epoch: 9000 Loss: 0.007743935566395521
Epoch: 9001 Loss: 0.007742641493678093
Epoch: 9002 Loss: 0.007740445900708437
Epoch: 9003 Loss: 0.007739802356809378
Epoch: 9004 Loss: 0.007738783489912748
Epoch: 9005 Loss: 0.007737466134130955
Epoch: 9006 Loss: 0.007735428400337696
Epoch: 9007 Loss: 0.007734396029263735
Epoch: 9008 Loss: 0.007732389494776726
Epoch: 9009 Loss: 0.007731714751571417
Epoch: 9010 Loss: 0.00773039972409606
Epoch: 9011 Loss: 0.007729359902441502
Epoch: 9012 Loss: 0.007727597374469042
Epoch: 9013 Loss: 0.007725912611931562
Epoch: 9014 Loss: 0.007724542170763016
Epoch: 9015 Loss: 0.007723283022642136
Epoch: 9016 Loss: 0.007722882553935051
Epoch: 9017 Loss: 0.007720662280917168
Epoch: 9018 Loss: 0.007719694636762142
Epoch: 9019 Loss: 0.007718642707914114
Epoch: 9020 Loss: 0.007717208005487919
Epoch: 9021 Loss: 0.0077157518826425076
Epoch: 9022 Loss: 0.007714164908975363
Epoch: 9023 Loss: 0.007712657563388348
Epoch: 9024 Loss: 0.007711552549153566
Epoch: 9025 Loss: 0.007710189092904329
Epoch: 9026 Loss: 0.00770987756550312
Epoch: 9027 Loss: 0.007708927616477013
Epoch: 9028 Loss: 0.007706099655479193
Epoch: 9029 Loss: 0.007705487310886383
Epoch: 9030 Loss: 0.007704448886215687
Epoch: 9031 Loss: 0.007703200448304415
Epoch: 9032 Loss: 0.007701112423092127
Epoch: 9033 Loss: 0.007700176909565926
Epoch: 9034 Loss: 0.007698293775320053
Epoch: 9035 Loss: 0.007696620654314756
Epoch: 9036 Loss: 0.007695967331528664
Epoch: 9037 Loss: 0.007693772204220295
Epoch: 9038 Loss: 0.007693074643611908
Epoch: 9039 Loss: 0.0076917060650885105
Epoch: 9040 Loss: 0.0076918331906199455
Epoch: 9041 Loss: 0.0076892562210559845
Epoch: 9042 Loss: 0.007687882054597139
Epoch: 9043 Loss: 0.007685740943998098
Epoch: 9044 Loss: 0.007685492280870676
Epoch: 9045 Loss: 0.007684772368520498
Epoch: 9046 Loss: 0.007682958152145147
Epoch: 9047 Loss: 0.007681116461753845
Epoch: 9048 Loss: 0.007679713889956474
Epoch: 9049 Loss: 0.007678840309381485
Epoch: 9050 Loss: 0.007677065208554268
Epoch: 9051 Loss: 0.007675332482904196
Epoch: 9052 Loss: 0.007674280554056168
Epoch: 9053 Loss: 0.007673288229852915
Epoch: 9054 Loss: 0.007673179265111685
Epoch: 9055 Loss: 0.007670796476304531
Epoch: 9056 Loss: 0.007669775281101465
Epoch: 9057 Loss: 0.007668390404433012
Epoch: 9058 Loss: 0.007667118217796087
Epoch: 9059 Loss: 0.007665816694498062
Epoch: 9060 Loss: 0.0076649305410683155
Epoch: 9061 Loss: 0.007663583382964134
Epoch: 9062 Loss: 0.007661638781428337
Epoch: 9063 Loss: 0.007659875322133303
Epoch: 9064 Loss: 0.007659031078219414
Epoch: 9065 Loss: 0.007657674141228199
Epoch: 9066 Loss: 0.0076561677269637585
Epoch: 9067 Loss: 0.007654472719877958
Epoch: 9068 Loss: 0.007654109038412571
Epoch: 9069 Loss: 0.007652482483536005
Epoch: 9070 Loss: 0.007651749532669783
Epoch: 9071 Loss: 0.007649656385183334
Epoch: 9072 Loss: 0.0076477741822600365
Epoch: 9073 Loss: 0.007646583952009678
Epoch: 9074 Loss: 0.007644496392458677
Epoch: 9075 Loss: 0.007644011173397303
Epoch: 9076 Loss: 0.007642373442649841
Epoch: 9077 Loss: 0.00764133594930172
Epoch: 9078 Loss: 0.007639520335942507
Epoch: 9079 Loss: 0.007640166208148003
Epoch: 9080 Loss: 0.0076381550170481205
Epoch: 9081 Loss: 0.007636571303009987
Epoch: 9082 Loss: 0.007635436486452818
Epoch: 9083 Loss: 0.007634219713509083
Epoch: 9084 Loss: 0.00763207720592618
Epoch: 9085 Loss: 0.007630344480276108
Epoch: 9086 Loss: 0.007629497908055782
Epoch: 9087 Loss: 0.007628516294062138
Epoch: 9088 Loss: 0.007626766804605722
Epoch: 9089 Loss: 0.007626480422914028
Epoch: 9090 Loss: 0.007623747922480106
Epoch: 9091 Loss: 0.007623284589499235
Epoch: 9092 Loss: 0.007622810546308756
Epoch: 9093 Loss: 0.007621509023010731
Epoch: 9094 Loss: 0.0076201362535357475
Epoch: 9095 Loss: 0.007617684546858072
Epoch: 9096 Loss: 0.007616954389959574
Epoch: 9097 Loss: 0.007615169044584036
Epoch: 9098 Loss: 0.007613202091306448
Epoch: 9099 Loss: 0.007613166235387325
Epoch: 9100 Loss: 0.007611107546836138
Epoch: 9101 Loss: 0.007609567139297724
Epoch: 9102 Loss: 0.007608421146869659
Epoch: 9103 Loss: 0.0076071289367973804
Epoch: 9104 Loss: 0.00760597875341773
Epoch: 9105 Loss: 0.0076045081950724125
Epoch: 9106 Loss: 0.007603271398693323
Epoch: 9107 Loss: 0.007602210156619549
Epoch: 9108 Loss: 0.007600552402436733
Epoch: 9109 Loss: 0.00759951351210475
Epoch: 9110 Loss: 0.007597814314067364
Epoch: 9111 Loss: 0.007596617564558983
Epoch: 9112 Loss: 0.007594467606395483
Epoch: 9113 Loss: 0.007594965863972902
Epoch: 9114 Loss: 0.00759277306497097
Epoch: 9115 Loss: 0.0075926450081169605
Epoch: 9116 Loss: 0.007589796558022499
Epoch: 9117 Loss: 0.007588979322463274
Epoch: 9118 Loss: 0.0075875683687627316
Epoch: 9119 Loss: 0.007585626095533371
Epoch: 9120 Loss: 0.007584966719150543
Epoch: 9121 Loss: 0.007584544364362955
Epoch: 9122 Loss: 0.007582944817841053
Epoch: 9123 Loss: 0.007581933867186308
Epoch: 9124 Loss: 0.007579091936349869
Epoch: 9125 Loss: 0.007578776217997074
Epoch: 9126 Loss: 0.007577661890536547
Epoch: 9127 Loss: 0.007575910072773695
Epoch: 9128 Loss: 0.007574695628136396
Epoch: 9129 Loss: 0.00757388211786747
Epoch: 9130 Loss: 0.007572519592940807
Epoch: 9131 Loss: 0.007570780348032713
Epoch: 9132 Loss: 0.007569838315248489
Epoch: 9133 Loss: 0.0075675626285374165
Epoch: 9134 Loss: 0.00756643433123827
Epoch: 9135 Loss: 0.007565720938146114
Epoch: 9136 Loss: 0.007564314641058445
Epoch: 9137 Loss: 0.007562572136521339
Epoch: 9138 Loss: 0.007562367245554924
Epoch: 9139 Loss: 0.007560069672763348
Epoch: 9140 Loss: 0.007559165824204683
Epoch: 9141 Loss: 0.00755776884034276
Epoch: 9142 Loss: 0.00755665497854352
Epoch: 9143 Loss: 0.007554762996733189
Epoch: 9144 Loss: 0.007553267292678356
Epoch: 9145 Loss: 0.007552688475698233
Epoch: 9146 Loss: 0.0075505562126636505
Epoch: 9147 Loss: 0.007550311740487814
Epoch: 9148 Loss: 0.007549359928816557
Epoch: 9149 Loss: 0.007548010908067226
Epoch: 9150 Loss: 0.0075452919118106365
Epoch: 9151 Loss: 0.0075444323010742664
Epoch: 9152 Loss: 0.00754436943680048
Epoch: 9153 Loss: 0.007542881183326244
Epoch: 9154 Loss: 0.007541671860963106
Epoch: 9155 Loss: 0.007539710961282253
Epoch: 9156 Loss: 0.007537875324487686
Epoch: 9157 Loss: 0.007536909077316523
Epoch: 9158 Loss: 0.007535959128290415
Epoch: 9159 Loss: 0.007534031756222248
Epoch: 9160 Loss: 0.0075336117297410965
Epoch: 9161 Loss: 0.007532489486038685
Epoch: 9162 Loss: 0.007531137205660343
Epoch: 9163 Loss: 0.007529460359364748
Epoch: 9164 Loss: 0.007527567446231842
Epoch: 9165 Loss: 0.007527269888669252
Epoch: 9166 Loss: 0.007525645196437836
Epoch: 9167 Loss: 0.007524356245994568
Epoch: 9168 Loss: 0.007523718755692244
Epoch: 9169 Loss: 0.007522146683186293
Epoch: 9170 Loss: 0.00752061465755105
Epoch: 9171 Loss: 0.007519100327044725
Epoch: 9172 Loss: 0.0075178248807787895
Epoch: 9173 Loss: 0.007516349200159311
Epoch: 9174 Loss: 0.007515518926084042
Epoch: 9175 Loss: 0.0075144642032682896
Epoch: 9176 Loss: 0.00751293171197176
Epoch: 9177 Loss: 0.0075117615051567554
Epoch: 9178 Loss: 0.007509683258831501
Epoch: 9179 Loss: 0.007509100716561079
Epoch: 9180 Loss: 0.007506901398301125
Epoch: 9181 Loss: 0.007506911642849445
Epoch: 9182 Loss: 0.007505019195377827
Epoch: 9183 Loss: 0.007503989618271589
Epoch: 9184 Loss: 0.007502985652536154
Epoch: 9185 Loss: 0.007501176092773676
Epoch: 9186 Loss: 0.007499866187572479
Epoch: 9187 Loss: 0.007499184925109148
Epoch: 9188 Loss: 0.007497601676732302
Epoch: 9189 Loss: 0.007496102713048458
Epoch: 9190 Loss: 0.007495581172406673
Epoch: 9191 Loss: 0.00749420840293169
Epoch: 9192 Loss: 0.00749237323179841
Epoch: 9193 Loss: 0.007491246797144413
Epoch: 9194 Loss: 0.007489501498639584
Epoch: 9195 Loss: 0.0074887461960315704
Epoch: 9196 Loss: 0.007487827911973
Epoch: 9197 Loss: 0.007486098911613226
Epoch: 9198 Loss: 0.007484549190849066
Epoch: 9199 Loss: 0.0074843792244791985
Epoch: 9200 Loss: 0.007482074201107025
Epoch: 9201 Loss: 0.007481690030544996
Epoch: 9202 Loss: 0.007480282802134752
Epoch: 9203 Loss: 0.007479185704141855
Epoch: 9204 Loss: 0.007477705366909504
Epoch: 9205 Loss: 0.007476507220417261
Epoch: 9206 Loss: 0.007474074140191078
Epoch: 9207 Loss: 0.007472931407392025
Epoch: 9208 Loss: 0.007472222205251455
Epoch: 9209 Loss: 0.0074713812209665775
Epoch: 9210 Loss: 0.007470217067748308
Epoch: 9211 Loss: 0.007468864321708679
Epoch: 9212 Loss: 0.007467426359653473
Epoch: 9213 Loss: 0.007466012146323919
Epoch: 9214 Loss: 0.007464962545782328
Epoch: 9215 Loss: 0.0074633024632930756
Epoch: 9216 Loss: 0.007462462410330772
Epoch: 9217 Loss: 0.00746132293716073
Epoch: 9218 Loss: 0.007460274267941713
Epoch: 9219 Loss: 0.007458241190761328
Epoch: 9220 Loss: 0.007456681691110134
Epoch: 9221 Loss: 0.007455621846020222
Epoch: 9222 Loss: 0.0074546500109136105
Epoch: 9223 Loss: 0.007452788762748241
Epoch: 9224 Loss: 0.007451874669641256
Epoch: 9225 Loss: 0.007450464181602001
Epoch: 9226 Loss: 0.007450917270034552
Epoch: 9227 Loss: 0.007449348457157612
Epoch: 9228 Loss: 0.007447666488587856
Epoch: 9229 Loss: 0.007445966824889183
Epoch: 9230 Loss: 0.007444476243108511
Epoch: 9231 Loss: 0.007442960515618324
Epoch: 9232 Loss: 0.007442348171025515
Epoch: 9233 Loss: 0.007440609857439995
Epoch: 9234 Loss: 0.0074399309232831
Epoch: 9235 Loss: 0.007437861058861017
Epoch: 9236 Loss: 0.007437015883624554
Epoch: 9237 Loss: 0.0074359457939863205
Epoch: 9238 Loss: 0.007434907369315624
Epoch: 9239 Loss: 0.007433025632053614
Epoch: 9240 Loss: 0.007432372774928808
Epoch: 9241 Loss: 0.00743055809289217
Epoch: 9242 Loss: 0.007429471705108881
Epoch: 9243 Loss: 0.007428507786244154
Epoch: 9244 Loss: 0.007427537813782692
Epoch: 9245 Loss: 0.007425726857036352
Epoch: 9246 Loss: 0.007424919866025448
Epoch: 9247 Loss: 0.0074230702593922615
Epoch: 9248 Loss: 0.007421841844916344
Epoch: 9249 Loss: 0.007420800160616636
Epoch: 9250 Loss: 0.007419714704155922
Epoch: 9251 Loss: 0.007418560795485973
Epoch: 9252 Loss: 0.007417956832796335
Epoch: 9253 Loss: 0.007416444830596447
Epoch: 9254 Loss: 0.0074141742661595345
Epoch: 9255 Loss: 0.007413579151034355
Epoch: 9256 Loss: 0.007412035949528217
Epoch: 9257 Loss: 0.007410992868244648
Epoch: 9258 Loss: 0.007409930229187012
Epoch: 9259 Loss: 0.0074083865620195866
Epoch: 9260 Loss: 0.007407776545733213
Epoch: 9261 Loss: 0.0074058049358427525
Epoch: 9262 Loss: 0.007404766511172056
Epoch: 9263 Loss: 0.00740280793979764
Epoch: 9264 Loss: 0.007402394432574511
Epoch: 9265 Loss: 0.007401768583804369
Epoch: 9266 Loss: 0.007400283124297857
Epoch: 9267 Loss: 0.00739884190261364
Epoch: 9268 Loss: 0.007396939676254988
Epoch: 9269 Loss: 0.007396411150693893
Epoch: 9270 Loss: 0.007394821848720312
Epoch: 9271 Loss: 0.007394242566078901
Epoch: 9272 Loss: 0.007392285391688347
Epoch: 9273 Loss: 0.007390946615487337
Epoch: 9274 Loss: 0.007389322854578495
Epoch: 9275 Loss: 0.00738925114274025
Epoch: 9276 Loss: 0.007387148682028055
Epoch: 9277 Loss: 0.007386215031147003
Epoch: 9278 Loss: 0.007385046686977148
Epoch: 9279 Loss: 0.007383911870419979
Epoch: 9280 Loss: 0.007383507676422596
Epoch: 9281 Loss: 0.007380823604762554
Epoch: 9282 Loss: 0.007380335591733456
Epoch: 9283 Loss: 0.007379662711173296
Epoch: 9284 Loss: 0.0073775299824774265
Epoch: 9285 Loss: 0.0073757353238761425
Epoch: 9286 Loss: 0.007375734392553568
Epoch: 9287 Loss: 0.00737401656806469
Epoch: 9288 Loss: 0.007373189553618431
Epoch: 9289 Loss: 0.007371046580374241
Epoch: 9290 Loss: 0.007370258215814829
Epoch: 9291 Loss: 0.007368825376033783
Epoch: 9292 Loss: 0.007367243058979511
Epoch: 9293 Loss: 0.007367810700088739
Epoch: 9294 Loss: 0.007365860510617495
Epoch: 9295 Loss: 0.007364184595644474
Epoch: 9296 Loss: 0.007362975738942623
Epoch: 9297 Loss: 0.007361701223999262
Epoch: 9298 Loss: 0.0073607745580375195
Epoch: 9299 Loss: 0.007359836250543594
Epoch: 9300 Loss: 0.007358225528150797
Epoch: 9301 Loss: 0.0073564015328884125
Epoch: 9302 Loss: 0.007356386631727219
Epoch: 9303 Loss: 0.007354534696787596
Epoch: 9304 Loss: 0.007352739106863737
Epoch: 9305 Loss: 0.007351751439273357
Epoch: 9306 Loss: 0.007351137697696686
Epoch: 9307 Loss: 0.007349973078817129
Epoch: 9308 Loss: 0.00734849413856864
Epoch: 9309 Loss: 0.007347224745899439
Epoch: 9310 Loss: 0.0073461527936160564
Epoch: 9311 Loss: 0.007343692239373922
Epoch: 9312 Loss: 0.007343853823840618
Epoch: 9313 Loss: 0.00734298350289464
Epoch: 9314 Loss: 0.007340663578361273
Epoch: 9315 Loss: 0.0073392149060964584
Epoch: 9316 Loss: 0.007338159717619419
Epoch: 9317 Loss: 0.007338013034313917
Epoch: 9318 Loss: 0.0073360842652618885
Epoch: 9319 Loss: 0.007334819063544273
Epoch: 9320 Loss: 0.007333044428378344
Epoch: 9321 Loss: 0.0073331729508936405
Epoch: 9322 Loss: 0.007331172004342079
Epoch: 9323 Loss: 0.007329522166401148
Epoch: 9324 Loss: 0.007328789681196213
Epoch: 9325 Loss: 0.007328189443796873
Epoch: 9326 Loss: 0.0073257493786513805
Epoch: 9327 Loss: 0.00732509745284915
Epoch: 9328 Loss: 0.007323930971324444
Epoch: 9329 Loss: 0.007322975434362888
Epoch: 9330 Loss: 0.007321779616177082
Epoch: 9331 Loss: 0.007320253644138575
Epoch: 9332 Loss: 0.007319591473788023
Epoch: 9333 Loss: 0.007317873649299145
Epoch: 9334 Loss: 0.0073163737542927265
Epoch: 9335 Loss: 0.007315405644476414
Epoch: 9336 Loss: 0.007314082235097885
Epoch: 9337 Loss: 0.007313299458473921
Epoch: 9338 Loss: 0.007311687804758549
Epoch: 9339 Loss: 0.007311537396162748
Epoch: 9340 Loss: 0.007309942971915007
Epoch: 9341 Loss: 0.007309190463274717
Epoch: 9342 Loss: 0.007307285908609629
Epoch: 9343 Loss: 0.007306703366339207
Epoch: 9344 Loss: 0.007304418832063675
Epoch: 9345 Loss: 0.007303649093955755
Epoch: 9346 Loss: 0.007301959674805403
Epoch: 9347 Loss: 0.0073005747981369495
Epoch: 9348 Loss: 0.007299453020095825
Epoch: 9349 Loss: 0.007298657204955816
Epoch: 9350 Loss: 0.007297194562852383
Epoch: 9351 Loss: 0.007296649273484945
Epoch: 9352 Loss: 0.007295035291463137
Epoch: 9353 Loss: 0.0072938124649226665
Epoch: 9354 Loss: 0.007292135618627071
Epoch: 9355 Loss: 0.007290774490684271
Epoch: 9356 Loss: 0.007290182169526815
Epoch: 9357 Loss: 0.00728940311819315
Epoch: 9358 Loss: 0.0072878263890743256
Epoch: 9359 Loss: 0.007287567015737295
Epoch: 9360 Loss: 0.0072857169434428215
Epoch: 9361 Loss: 0.007284610997885466
Epoch: 9362 Loss: 0.007282933685928583
Epoch: 9363 Loss: 0.007282198406755924
Epoch: 9364 Loss: 0.007281347643584013
Epoch: 9365 Loss: 0.007279698736965656
Epoch: 9366 Loss: 0.0072789303958415985
Epoch: 9367 Loss: 0.007277136668562889
Epoch: 9368 Loss: 0.007275484036654234
Epoch: 9369 Loss: 0.007274510804563761
Epoch: 9370 Loss: 0.007273640017956495
Epoch: 9371 Loss: 0.0072718155570328236
Epoch: 9372 Loss: 0.007271502632647753
Epoch: 9373 Loss: 0.00727034080773592
Epoch: 9374 Loss: 0.007268861867487431
Epoch: 9375 Loss: 0.007268104702234268
Epoch: 9376 Loss: 0.007266508415341377
Epoch: 9377 Loss: 0.00726493401452899
Epoch: 9378 Loss: 0.007263682316988707
Epoch: 9379 Loss: 0.0072634038515388966
Epoch: 9380 Loss: 0.007262367755174637
Epoch: 9381 Loss: 0.007260418962687254
Epoch: 9382 Loss: 0.007259176578372717
Epoch: 9383 Loss: 0.0072580622509121895
Epoch: 9384 Loss: 0.007257419638335705
Epoch: 9385 Loss: 0.007255639880895615
Epoch: 9386 Loss: 0.007255065254867077
Epoch: 9387 Loss: 0.007252988871186972
Epoch: 9388 Loss: 0.007252517621964216
Epoch: 9389 Loss: 0.0072508761659264565
Epoch: 9390 Loss: 0.007249878253787756
Epoch: 9391 Loss: 0.007248298730701208
Epoch: 9392 Loss: 0.007248142268508673
Epoch: 9393 Loss: 0.007246604189276695
Epoch: 9394 Loss: 0.007245011627674103
Epoch: 9395 Loss: 0.007243604864925146
Epoch: 9396 Loss: 0.007242667023092508
Epoch: 9397 Loss: 0.007241284940391779
Epoch: 9398 Loss: 0.007241036742925644
Epoch: 9399 Loss: 0.007238919846713543
Epoch: 9400 Loss: 0.007237447425723076
Epoch: 9401 Loss: 0.007237079553306103
Epoch: 9402 Loss: 0.007236751262098551
Epoch: 9403 Loss: 0.007235122844576836
Epoch: 9404 Loss: 0.007233386859297752
Epoch: 9405 Loss: 0.007232110947370529
Epoch: 9406 Loss: 0.007230667397379875
Epoch: 9407 Loss: 0.007229248993098736
Epoch: 9408 Loss: 0.007228474598377943
Epoch: 9409 Loss: 0.00722784548997879
Epoch: 9410 Loss: 0.007226372137665749
Epoch: 9411 Loss: 0.007225191220641136
Epoch: 9412 Loss: 0.007223856169730425
Epoch: 9413 Loss: 0.0072228265926241875
Epoch: 9414 Loss: 0.007221330888569355
Epoch: 9415 Loss: 0.007220057770609856
Epoch: 9416 Loss: 0.0072189499624073505
Epoch: 9417 Loss: 0.007217949256300926
Epoch: 9418 Loss: 0.007216665893793106
Epoch: 9419 Loss: 0.007215579506009817
Epoch: 9420 Loss: 0.007215152494609356
Epoch: 9421 Loss: 0.00721299322322011
Epoch: 9422 Loss: 0.007212613243609667
Epoch: 9423 Loss: 0.00721094710752368
Epoch: 9424 Loss: 0.007209244649857283
Epoch: 9425 Loss: 0.0072077801451087
Epoch: 9426 Loss: 0.007208334282040596
Epoch: 9427 Loss: 0.00720607116818428
Epoch: 9428 Loss: 0.00720421364530921
Epoch: 9429 Loss: 0.007203984539955854
Epoch: 9430 Loss: 0.007203214336186647
Epoch: 9431 Loss: 0.007201348897069693
Epoch: 9432 Loss: 0.007200787775218487
Epoch: 9433 Loss: 0.007198922336101532
Epoch: 9434 Loss: 0.007198479492217302
Epoch: 9435 Loss: 0.007196689955890179
Epoch: 9436 Loss: 0.0071952384896576405
Epoch: 9437 Loss: 0.0071945092640817165
Epoch: 9438 Loss: 0.007194267585873604
Epoch: 9439 Loss: 0.0071922303177416325
Epoch: 9440 Loss: 0.007191497832536697
Epoch: 9441 Loss: 0.00718989223241806
Epoch: 9442 Loss: 0.007188815623521805
Epoch: 9443 Loss: 0.007187000010162592
Epoch: 9444 Loss: 0.007186263334006071
Epoch: 9445 Loss: 0.007184892427176237
Epoch: 9446 Loss: 0.007184505462646484
Epoch: 9447 Loss: 0.007182561792433262
Epoch: 9448 Loss: 0.007182687520980835
Epoch: 9449 Loss: 0.0071805198676884174
Epoch: 9450 Loss: 0.007179868873208761
Epoch: 9451 Loss: 0.007178769446909428
Epoch: 9452 Loss: 0.007177484687417746
Epoch: 9453 Loss: 0.007175781764090061
Epoch: 9454 Loss: 0.007174397818744183
Epoch: 9455 Loss: 0.007173489313572645
Epoch: 9456 Loss: 0.007172605488449335
Epoch: 9457 Loss: 0.007171737961471081
Epoch: 9458 Loss: 0.007169825956225395
Epoch: 9459 Loss: 0.00716854864731431
Epoch: 9460 Loss: 0.007168234791606665
Epoch: 9461 Loss: 0.007166786585003138
Epoch: 9462 Loss: 0.007165543269366026
Epoch: 9463 Loss: 0.007164330687373877
Epoch: 9464 Loss: 0.007162983529269695
Epoch: 9465 Loss: 0.007162085734307766
Epoch: 9466 Loss: 0.007160989101976156
Epoch: 9467 Loss: 0.007159637287259102
Epoch: 9468 Loss: 0.007158803287893534
Epoch: 9469 Loss: 0.007157732732594013
Epoch: 9470 Loss: 0.007156022824347019
Epoch: 9471 Loss: 0.007154782302677631
Epoch: 9472 Loss: 0.007154013495892286
Epoch: 9473 Loss: 0.007153340149670839
Epoch: 9474 Loss: 0.007152080535888672
Epoch: 9475 Loss: 0.007150675635784864
Epoch: 9476 Loss: 0.007148966658860445
Epoch: 9477 Loss: 0.0071478537283837795
Epoch: 9478 Loss: 0.007146638818085194
Epoch: 9479 Loss: 0.007146036718040705
Epoch: 9480 Loss: 0.007144316099584103
Epoch: 9481 Loss: 0.007143072318285704
Epoch: 9482 Loss: 0.00714246416464448
Epoch: 9483 Loss: 0.007141825743019581
Epoch: 9484 Loss: 0.007140519097447395
Epoch: 9485 Loss: 0.007138880435377359
Epoch: 9486 Loss: 0.007138486485928297
Epoch: 9487 Loss: 0.007136452477425337
Epoch: 9488 Loss: 0.0071356575936079025
Epoch: 9489 Loss: 0.007134042214602232
Epoch: 9490 Loss: 0.007133900187909603
Epoch: 9491 Loss: 0.007131502497941256
Epoch: 9492 Loss: 0.0071305399760603905
Epoch: 9493 Loss: 0.007129592355340719
Epoch: 9494 Loss: 0.007129072677344084
Epoch: 9495 Loss: 0.007127992808818817
Epoch: 9496 Loss: 0.007126501761376858
Epoch: 9497 Loss: 0.007125185802578926
Epoch: 9498 Loss: 0.007123306859284639
Epoch: 9499 Loss: 0.007123145274817944
Epoch: 9500 Loss: 0.007122081238776445
Epoch: 9501 Loss: 0.0071206665597856045
Epoch: 9502 Loss: 0.007119069807231426
Epoch: 9503 Loss: 0.007118492387235165
Epoch: 9504 Loss: 0.007116987369954586
Epoch: 9505 Loss: 0.007115407381206751
Epoch: 9506 Loss: 0.007115162443369627
Epoch: 9507 Loss: 0.0071138353087008
Epoch: 9508 Loss: 0.007112567313015461
Epoch: 9509 Loss: 0.0071112471632659435
Epoch: 9510 Loss: 0.007110242731869221
Epoch: 9511 Loss: 0.007109414786100388
Epoch: 9512 Loss: 0.0071076080203056335
Epoch: 9513 Loss: 0.007106835953891277
Epoch: 9514 Loss: 0.00710573885589838
Epoch: 9515 Loss: 0.007104603573679924
Epoch: 9516 Loss: 0.007104018237441778
Epoch: 9517 Loss: 0.007102212402969599
Epoch: 9518 Loss: 0.007101910188794136
Epoch: 9519 Loss: 0.007100515533238649
Epoch: 9520 Loss: 0.007099424488842487
Epoch: 9521 Loss: 0.007097545545548201
Epoch: 9522 Loss: 0.007096433080732822
Epoch: 9523 Loss: 0.007095342501997948
Epoch: 9524 Loss: 0.007093614898622036
Epoch: 9525 Loss: 0.00709253316745162
Epoch: 9526 Loss: 0.00709257647395134
Epoch: 9527 Loss: 0.0070906272158026695
Epoch: 9528 Loss: 0.007089927326887846
Epoch: 9529 Loss: 0.007088871672749519
Epoch: 9530 Loss: 0.007088267710059881
Epoch: 9531 Loss: 0.007086662575602531
Epoch: 9532 Loss: 0.007086183410137892
Epoch: 9533 Loss: 0.007084414828568697
Epoch: 9534 Loss: 0.007082833908498287
Epoch: 9535 Loss: 0.007082057651132345
Epoch: 9536 Loss: 0.007081137504428625
Epoch: 9537 Loss: 0.007078767288476229
Epoch: 9538 Loss: 0.007078469730913639
Epoch: 9539 Loss: 0.007077558431774378
Epoch: 9540 Loss: 0.007076764013618231
Epoch: 9541 Loss: 0.007075252011418343
Epoch: 9542 Loss: 0.007073337212204933
Epoch: 9543 Loss: 0.0070729246363043785
Epoch: 9544 Loss: 0.007071597967296839
Epoch: 9545 Loss: 0.0070708515122532845
Epoch: 9546 Loss: 0.007069774437695742
Epoch: 9547 Loss: 0.007067983504384756
Epoch: 9548 Loss: 0.007068405859172344
Epoch: 9549 Loss: 0.007066055200994015
Epoch: 9550 Loss: 0.00706513412296772
Epoch: 9551 Loss: 0.00706342002376914
Epoch: 9552 Loss: 0.007062367629259825
Epoch: 9553 Loss: 0.007062068674713373
Epoch: 9554 Loss: 0.007060796953737736
Epoch: 9555 Loss: 0.007059517782181501
Epoch: 9556 Loss: 0.007057825569063425
Epoch: 9557 Loss: 0.007056733593344688
Epoch: 9558 Loss: 0.007055709604173899
Epoch: 9559 Loss: 0.0070554702542722225
Epoch: 9560 Loss: 0.007053667679429054
Epoch: 9561 Loss: 0.0070520685985684395
Epoch: 9562 Loss: 0.007051075808703899
Epoch: 9563 Loss: 0.007050592917948961
Epoch: 9564 Loss: 0.007049566600471735
Epoch: 9565 Loss: 0.007047944236546755
Epoch: 9566 Loss: 0.007047200575470924
Epoch: 9567 Loss: 0.007045628037303686
Epoch: 9568 Loss: 0.007044179830700159
Epoch: 9569 Loss: 0.007044006604701281
Epoch: 9570 Loss: 0.007042525801807642
Epoch: 9571 Loss: 0.007041780278086662
Epoch: 9572 Loss: 0.00704016350209713
Epoch: 9573 Loss: 0.007038560695946217
Epoch: 9574 Loss: 0.007037946954369545
Epoch: 9575 Loss: 0.007036574184894562
Epoch: 9576 Loss: 0.007036241237074137
Epoch: 9577 Loss: 0.007035014219582081
Epoch: 9578 Loss: 0.007033096626400948
Epoch: 9579 Loss: 0.0070329513400793076
Epoch: 9580 Loss: 0.007031251210719347
Epoch: 9581 Loss: 0.007029367610812187
Epoch: 9582 Loss: 0.0070297615602612495
Epoch: 9583 Loss: 0.007028210908174515
Epoch: 9584 Loss: 0.0070267412811517715
Epoch: 9585 Loss: 0.007025479804724455
Epoch: 9586 Loss: 0.007024242077022791
Epoch: 9587 Loss: 0.007023269776254892
Epoch: 9588 Loss: 0.0070219142362475395
Epoch: 9589 Loss: 0.0070213694125413895
Epoch: 9590 Loss: 0.007020903751254082
Epoch: 9591 Loss: 0.0070188650861382484
Epoch: 9592 Loss: 0.007017952855676413
Epoch: 9593 Loss: 0.007016455754637718
Epoch: 9594 Loss: 0.007016282994300127
Epoch: 9595 Loss: 0.007014899514615536
Epoch: 9596 Loss: 0.00701337493956089
Epoch: 9597 Loss: 0.007012408226728439
Epoch: 9598 Loss: 0.007010836619883776
Epoch: 9599 Loss: 0.0070097860880196095
Epoch: 9600 Loss: 0.0070090219378471375
Epoch: 9601 Loss: 0.00700780563056469
Epoch: 9602 Loss: 0.007006500381976366
Epoch: 9603 Loss: 0.00700661214068532
Epoch: 9604 Loss: 0.007004861254245043
Epoch: 9605 Loss: 0.007003271020948887
Epoch: 9606 Loss: 0.007002136204391718
Epoch: 9607 Loss: 0.007001159247010946
Epoch: 9608 Loss: 0.006999448873102665
Epoch: 9609 Loss: 0.006998722441494465
Epoch: 9610 Loss: 0.006997378543019295
Epoch: 9611 Loss: 0.0069963629357516766
Epoch: 9612 Loss: 0.006996316369622946
Epoch: 9613 Loss: 0.0069944728165864944
Epoch: 9614 Loss: 0.00699369702488184
Epoch: 9615 Loss: 0.006992735899984837
Epoch: 9616 Loss: 0.006991386413574219
Epoch: 9617 Loss: 0.006989866029471159
Epoch: 9618 Loss: 0.006989056244492531
Epoch: 9619 Loss: 0.0069875568151474
Epoch: 9620 Loss: 0.0069865998812019825
Epoch: 9621 Loss: 0.006985458079725504
Epoch: 9622 Loss: 0.006985087879002094
Epoch: 9623 Loss: 0.006983581930398941
Epoch: 9624 Loss: 0.006982196122407913
Epoch: 9625 Loss: 0.006981045939028263
Epoch: 9626 Loss: 0.006980090402066708
Epoch: 9627 Loss: 0.006979428231716156
Epoch: 9628 Loss: 0.0069779749028384686
Epoch: 9629 Loss: 0.00697648199275136
Epoch: 9630 Loss: 0.006976454984396696
Epoch: 9631 Loss: 0.006975235883146524
Epoch: 9632 Loss: 0.006973428651690483
Epoch: 9633 Loss: 0.006971854250878096
Epoch: 9634 Loss: 0.006971284281462431
Epoch: 9635 Loss: 0.0069700381718575954
Epoch: 9636 Loss: 0.006968370173126459
Epoch: 9637 Loss: 0.0069687096402049065
Epoch: 9638 Loss: 0.006967326160520315
Epoch: 9639 Loss: 0.006966233253479004
Epoch: 9640 Loss: 0.006963997147977352
Epoch: 9641 Loss: 0.006962865125387907
Epoch: 9642 Loss: 0.006962769664824009
Epoch: 9643 Loss: 0.006961651146411896
Epoch: 9644 Loss: 0.0069607398472726345
Epoch: 9645 Loss: 0.0069594248197972775
Epoch: 9646 Loss: 0.006958283018320799
Epoch: 9647 Loss: 0.006957331672310829
Epoch: 9648 Loss: 0.006956411059945822
Epoch: 9649 Loss: 0.006954723969101906
Epoch: 9650 Loss: 0.006953173782676458
Epoch: 9651 Loss: 0.006952167954295874
Epoch: 9652 Loss: 0.006951938383281231
Epoch: 9653 Loss: 0.006949795875698328
Epoch: 9654 Loss: 0.006950240582227707
Epoch: 9655 Loss: 0.006947710644453764
Epoch: 9656 Loss: 0.006946895271539688
Epoch: 9657 Loss: 0.006946016103029251
Epoch: 9658 Loss: 0.006945773959159851
Epoch: 9659 Loss: 0.006944588851183653
Epoch: 9660 Loss: 0.006943000480532646
Epoch: 9661 Loss: 0.0069412109442055225
Epoch: 9662 Loss: 0.006940758787095547
Epoch: 9663 Loss: 0.006939142942428589
Epoch: 9664 Loss: 0.0069389925338327885
Epoch: 9665 Loss: 0.006937125697731972
Epoch: 9666 Loss: 0.006936612073332071
Epoch: 9667 Loss: 0.006935047451406717
Epoch: 9668 Loss: 0.0069343410432338715
Epoch: 9669 Loss: 0.006932999938726425
Epoch: 9670 Loss: 0.006932743825018406
Epoch: 9671 Loss: 0.006931535433977842
Epoch: 9672 Loss: 0.006929650902748108
Epoch: 9673 Loss: 0.006928298156708479
Epoch: 9674 Loss: 0.0069275819696486
Epoch: 9675 Loss: 0.0069267298094928265
Epoch: 9676 Loss: 0.0069253770634531975
Epoch: 9677 Loss: 0.006923919543623924
Epoch: 9678 Loss: 0.006922668777406216
Epoch: 9679 Loss: 0.006922021508216858
Epoch: 9680 Loss: 0.006920777261257172
Epoch: 9681 Loss: 0.0069196950644254684
Epoch: 9682 Loss: 0.006919072009623051
Epoch: 9683 Loss: 0.006918488536030054
Epoch: 9684 Loss: 0.006916358135640621
Epoch: 9685 Loss: 0.006915802601724863
Epoch: 9686 Loss: 0.006914510857313871
Epoch: 9687 Loss: 0.006913588382303715
Epoch: 9688 Loss: 0.006912717595696449
Epoch: 9689 Loss: 0.006911144126206636
Epoch: 9690 Loss: 0.006909957155585289
Epoch: 9691 Loss: 0.00690867705270648
Epoch: 9692 Loss: 0.006907956674695015
Epoch: 9693 Loss: 0.0069067347794771194
Epoch: 9694 Loss: 0.0069056060165166855
Epoch: 9695 Loss: 0.006905026733875275
Epoch: 9696 Loss: 0.006903643719851971
Epoch: 9697 Loss: 0.006902429740875959
Epoch: 9698 Loss: 0.006902115885168314
Epoch: 9699 Loss: 0.00690067745745182
Epoch: 9700 Loss: 0.006899051368236542
Epoch: 9701 Loss: 0.006898487452417612
Epoch: 9702 Loss: 0.006897125393152237
Epoch: 9703 Loss: 0.006895688362419605
Epoch: 9704 Loss: 0.00689465319737792
Epoch: 9705 Loss: 0.006893533747643232
Epoch: 9706 Loss: 0.006892677396535873
Epoch: 9707 Loss: 0.006891200318932533
Epoch: 9708 Loss: 0.006890090182423592
Epoch: 9709 Loss: 0.006890198215842247
Epoch: 9710 Loss: 0.006888390518724918
Epoch: 9711 Loss: 0.006887425668537617
Epoch: 9712 Loss: 0.006886384449899197
Epoch: 9713 Loss: 0.006885204929858446
Epoch: 9714 Loss: 0.006884512025862932
Epoch: 9715 Loss: 0.006883649155497551
Epoch: 9716 Loss: 0.00688214460387826
Epoch: 9717 Loss: 0.006880739703774452
Epoch: 9718 Loss: 0.006879738997668028
Epoch: 9719 Loss: 0.006878821179270744
Epoch: 9720 Loss: 0.006877392530441284
Epoch: 9721 Loss: 0.006876194383949041
Epoch: 9722 Loss: 0.006875496357679367
Epoch: 9723 Loss: 0.006874482147395611
Epoch: 9724 Loss: 0.00687407935038209
Epoch: 9725 Loss: 0.006872289348393679
Epoch: 9726 Loss: 0.006870755925774574
Epoch: 9727 Loss: 0.00686996802687645
Epoch: 9728 Loss: 0.006869541946798563
Epoch: 9729 Loss: 0.00686771422624588
Epoch: 9730 Loss: 0.006867849733680487
Epoch: 9731 Loss: 0.006866121664643288
Epoch: 9732 Loss: 0.006864812225103378
Epoch: 9733 Loss: 0.006864029914140701
Epoch: 9734 Loss: 0.006863337941467762
Epoch: 9735 Loss: 0.006861411035060883
Epoch: 9736 Loss: 0.006860125809907913
Epoch: 9737 Loss: 0.006859840825200081
Epoch: 9738 Loss: 0.0068585933186113834
Epoch: 9739 Loss: 0.006857449188828468
Epoch: 9740 Loss: 0.00685650110244751
Epoch: 9741 Loss: 0.006855941843241453
Epoch: 9742 Loss: 0.006854578386992216
Epoch: 9743 Loss: 0.0068531823344528675
Epoch: 9744 Loss: 0.00685138488188386
Epoch: 9745 Loss: 0.006850856356322765
Epoch: 9746 Loss: 0.006849769968539476
Epoch: 9747 Loss: 0.00684924703091383
Epoch: 9748 Loss: 0.006848527584224939
Epoch: 9749 Loss: 0.0068470388650894165
Epoch: 9750 Loss: 0.006845621392130852
Epoch: 9751 Loss: 0.006845252588391304
Epoch: 9752 Loss: 0.006843140814453363
Epoch: 9753 Loss: 0.006842534523457289
Epoch: 9754 Loss: 0.006841667927801609
Epoch: 9755 Loss: 0.006839973386377096
Epoch: 9756 Loss: 0.006838981993496418
Epoch: 9757 Loss: 0.0068391114473342896
Epoch: 9758 Loss: 0.006836738903075457
Epoch: 9759 Loss: 0.006836095359176397
Epoch: 9760 Loss: 0.006835809908807278
Epoch: 9761 Loss: 0.006834073923528194
Epoch: 9762 Loss: 0.006833073683083057
Epoch: 9763 Loss: 0.006832126062363386
Epoch: 9764 Loss: 0.00683019170537591
Epoch: 9765 Loss: 0.006829371675848961
Epoch: 9766 Loss: 0.006828323472291231
Epoch: 9767 Loss: 0.006827172357589006
Epoch: 9768 Loss: 0.006826602853834629
Epoch: 9769 Loss: 0.00682549923658371
Epoch: 9770 Loss: 0.006824888288974762
Epoch: 9771 Loss: 0.006823603063821793
Epoch: 9772 Loss: 0.006822253577411175
Epoch: 9773 Loss: 0.0068215602077543736
Epoch: 9774 Loss: 0.006819930858910084
Epoch: 9775 Loss: 0.006819139234721661
Epoch: 9776 Loss: 0.006818187888711691
Epoch: 9777 Loss: 0.006817162502557039
Epoch: 9778 Loss: 0.006816449109464884
Epoch: 9779 Loss: 0.006814239546656609
Epoch: 9780 Loss: 0.006813716143369675
Epoch: 9781 Loss: 0.0068123359233140945
Epoch: 9782 Loss: 0.006811443716287613
Epoch: 9783 Loss: 0.006810641847550869
Epoch: 9784 Loss: 0.006809816230088472
Epoch: 9785 Loss: 0.006808592937886715
Epoch: 9786 Loss: 0.006807588506489992
Epoch: 9787 Loss: 0.006806030869483948
Epoch: 9788 Loss: 0.0068052965216338634
Epoch: 9789 Loss: 0.006804313510656357
Epoch: 9790 Loss: 0.006802906282246113
Epoch: 9791 Loss: 0.006802082061767578
Epoch: 9792 Loss: 0.006801066920161247
Epoch: 9793 Loss: 0.006800555624067783
Epoch: 9794 Loss: 0.0067986175417900085
Epoch: 9795 Loss: 0.006798030808568001
Epoch: 9796 Loss: 0.006797051057219505
Epoch: 9797 Loss: 0.006795244757086039
Epoch: 9798 Loss: 0.006794762797653675
Epoch: 9799 Loss: 0.006793617270886898
Epoch: 9800 Loss: 0.006792711094021797
Epoch: 9801 Loss: 0.0067915827967226505
Epoch: 9802 Loss: 0.0067900693975389
Epoch: 9803 Loss: 0.006788922473788261
Epoch: 9804 Loss: 0.006788554135710001
Epoch: 9805 Loss: 0.0067869191989302635
Epoch: 9806 Loss: 0.00678583700209856
Epoch: 9807 Loss: 0.00678517110645771
Epoch: 9808 Loss: 0.006784515921026468
Epoch: 9809 Loss: 0.006783320102840662
Epoch: 9810 Loss: 0.006781957112252712
Epoch: 9811 Loss: 0.006780948955565691
Epoch: 9812 Loss: 0.00678028492256999
Epoch: 9813 Loss: 0.006778953596949577
Epoch: 9814 Loss: 0.006778059061616659
Epoch: 9815 Loss: 0.006776977796107531
Epoch: 9816 Loss: 0.006775969173759222
Epoch: 9817 Loss: 0.006774797569960356
Epoch: 9818 Loss: 0.006773616652935743
Epoch: 9819 Loss: 0.006772791966795921
Epoch: 9820 Loss: 0.006771920248866081
Epoch: 9821 Loss: 0.006770229898393154
Epoch: 9822 Loss: 0.006769722793251276
Epoch: 9823 Loss: 0.006768758874386549
Epoch: 9824 Loss: 0.006767325568944216
Epoch: 9825 Loss: 0.006765658501535654
Epoch: 9826 Loss: 0.006765296217054129
Epoch: 9827 Loss: 0.006763983052223921
Epoch: 9828 Loss: 0.00676347129046917
Epoch: 9829 Loss: 0.006761616561561823
Epoch: 9830 Loss: 0.006761982571333647
Epoch: 9831 Loss: 0.006760699208825827
Epoch: 9832 Loss: 0.006758987903594971
Epoch: 9833 Loss: 0.006758048664778471
Epoch: 9834 Loss: 0.0067565967328846455
Epoch: 9835 Loss: 0.006755523849278688
Epoch: 9836 Loss: 0.006754897069185972
Epoch: 9837 Loss: 0.00675360718742013
Epoch: 9838 Loss: 0.006752370856702328
Epoch: 9839 Loss: 0.006752440240234137
Epoch: 9840 Loss: 0.006751448847353458
Epoch: 9841 Loss: 0.0067489114589989185
Epoch: 9842 Loss: 0.006748588290065527
Epoch: 9843 Loss: 0.006747650448232889
Epoch: 9844 Loss: 0.006746307946741581
Epoch: 9845 Loss: 0.006745119579136372
Epoch: 9846 Loss: 0.006744120270013809
Epoch: 9847 Loss: 0.006743974052369595
Epoch: 9848 Loss: 0.00674216216430068
Epoch: 9849 Loss: 0.006740962155163288
Epoch: 9850 Loss: 0.006740606389939785
Epoch: 9851 Loss: 0.006739214528352022
Epoch: 9852 Loss: 0.0067377290688455105
Epoch: 9853 Loss: 0.006737551186233759
Epoch: 9854 Loss: 0.00673634884878993
Epoch: 9855 Loss: 0.006734937895089388
Epoch: 9856 Loss: 0.006733928807079792
Epoch: 9857 Loss: 0.006732874549925327
Epoch: 9858 Loss: 0.006731965113431215
Epoch: 9859 Loss: 0.0067305490374565125
Epoch: 9860 Loss: 0.0067299441434443
Epoch: 9861 Loss: 0.006728796288371086
Epoch: 9862 Loss: 0.006727790925651789
Epoch: 9863 Loss: 0.0067263138480484486
Epoch: 9864 Loss: 0.006725816056132317
Epoch: 9865 Loss: 0.0067249177955091
Epoch: 9866 Loss: 0.006723372265696526
Epoch: 9867 Loss: 0.0067229559645056725
Epoch: 9868 Loss: 0.006721882149577141
Epoch: 9869 Loss: 0.006720809265971184
Epoch: 9870 Loss: 0.0067190323024988174
Epoch: 9871 Loss: 0.006718406919389963
Epoch: 9872 Loss: 0.0067175826989114285
Epoch: 9873 Loss: 0.00671671237796545
Epoch: 9874 Loss: 0.006715351715683937
Epoch: 9875 Loss: 0.006715033669024706
Epoch: 9876 Loss: 0.006713246926665306
Epoch: 9877 Loss: 0.006711920723319054
Epoch: 9878 Loss: 0.006711256690323353
Epoch: 9879 Loss: 0.006710078101605177
Epoch: 9880 Loss: 0.006708812899887562
Epoch: 9881 Loss: 0.006708272732794285
Epoch: 9882 Loss: 0.006706809625029564
Epoch: 9883 Loss: 0.006705815903842449
Epoch: 9884 Loss: 0.00670537818223238
Epoch: 9885 Loss: 0.006704004481434822
Epoch: 9886 Loss: 0.006702960468828678
Epoch: 9887 Loss: 0.006701917387545109
Epoch: 9888 Loss: 0.006701203528791666
Epoch: 9889 Loss: 0.006700289901345968
Epoch: 9890 Loss: 0.0066989329643547535
Epoch: 9891 Loss: 0.006697891745716333
Epoch: 9892 Loss: 0.0066967676393687725
Epoch: 9893 Loss: 0.006696203723549843
Epoch: 9894 Loss: 0.006694993935525417
Epoch: 9895 Loss: 0.006693447940051556
Epoch: 9896 Loss: 0.006692313589155674
Epoch: 9897 Loss: 0.0066920556128025055
Epoch: 9898 Loss: 0.006690377369523048
Epoch: 9899 Loss: 0.0066896299831569195
Epoch: 9900 Loss: 0.006688444875180721
Epoch: 9901 Loss: 0.006687333341687918
Epoch: 9902 Loss: 0.0066872211173176765
Epoch: 9903 Loss: 0.006685522384941578
Epoch: 9904 Loss: 0.0066851661540567875
Epoch: 9905 Loss: 0.006682992912828922
Epoch: 9906 Loss: 0.0066830869764089584
Epoch: 9907 Loss: 0.006681155879050493
Epoch: 9908 Loss: 0.006680898368358612
Epoch: 9909 Loss: 0.006678824778646231
Epoch: 9910 Loss: 0.006677982863038778
Epoch: 9911 Loss: 0.0066771917045116425
Epoch: 9912 Loss: 0.006676610559225082
Epoch: 9913 Loss: 0.00667567877098918
Epoch: 9914 Loss: 0.006674368400126696
Epoch: 9915 Loss: 0.006673543248325586
Epoch: 9916 Loss: 0.006672398187220097
Epoch: 9917 Loss: 0.00667161587625742
Epoch: 9918 Loss: 0.006670347414910793
Epoch: 9919 Loss: 0.006669086404144764
Epoch: 9920 Loss: 0.006667774170637131
Epoch: 9921 Loss: 0.006666747387498617
Epoch: 9922 Loss: 0.006665920373052359
Epoch: 9923 Loss: 0.006665047258138657
Epoch: 9924 Loss: 0.0066641769371926785
Epoch: 9925 Loss: 0.006662821862846613
Epoch: 9926 Loss: 0.006661649327725172
Epoch: 9927 Loss: 0.0066608707420527935
Epoch: 9928 Loss: 0.0066598448902368546
Epoch: 9929 Loss: 0.0066588264890015125
Epoch: 9930 Loss: 0.006657828111201525
Epoch: 9931 Loss: 0.006657086778432131
Epoch: 9932 Loss: 0.006656321231275797
Epoch: 9933 Loss: 0.00665486603975296
Epoch: 9934 Loss: 0.00665424345061183
Epoch: 9935 Loss: 0.00665275240316987
Epoch: 9936 Loss: 0.006651487201452255
Epoch: 9937 Loss: 0.006650265771895647
Epoch: 9938 Loss: 0.00665026530623436
Epoch: 9939 Loss: 0.006648398004472256
Epoch: 9940 Loss: 0.006647207774221897
Epoch: 9941 Loss: 0.006646882276982069
Epoch: 9942 Loss: 0.0066451639868319035
Epoch: 9943 Loss: 0.006644913926720619
Epoch: 9944 Loss: 0.006643820088356733
Epoch: 9945 Loss: 0.0066426824778318405
Epoch: 9946 Loss: 0.0066415113396942616
Epoch: 9947 Loss: 0.006640425883233547
Epoch: 9948 Loss: 0.006639369297772646
Epoch: 9949 Loss: 0.006638795603066683
Epoch: 9950 Loss: 0.006637359503656626
Epoch: 9951 Loss: 0.006636294536292553
Epoch: 9952 Loss: 0.006635462399572134
Epoch: 9953 Loss: 0.006634172052145004
Epoch: 9954 Loss: 0.006632943172007799
Epoch: 9955 Loss: 0.006632658187299967
Epoch: 9956 Loss: 0.00663127051666379
Epoch: 9957 Loss: 0.006631133612245321
Epoch: 9958 Loss: 0.006630649324506521
Epoch: 9959 Loss: 0.006628589238971472
Epoch: 9960 Loss: 0.006627520080655813
Epoch: 9961 Loss: 0.006626576650887728
Epoch: 9962 Loss: 0.00662568211555481
Epoch: 9963 Loss: 0.006623910274356604
Epoch: 9964 Loss: 0.006623247638344765
Epoch: 9965 Loss: 0.006622674874961376
Epoch: 9966 Loss: 0.006621215026825666
Epoch: 9967 Loss: 0.006620138883590698
Epoch: 9968 Loss: 0.00661957822740078
Epoch: 9969 Loss: 0.006618059705942869
Epoch: 9970 Loss: 0.0066172960214316845
Epoch: 9971 Loss: 0.006616067606955767
Epoch: 9972 Loss: 0.006614924408495426
Epoch: 9973 Loss: 0.006615237332880497
Epoch: 9974 Loss: 0.00661293463781476
Epoch: 9975 Loss: 0.0066118729300796986
Epoch: 9976 Loss: 0.006611463613808155
Epoch: 9977 Loss: 0.006610286422073841
Epoch: 9978 Loss: 0.006609508767724037
Epoch: 9979 Loss: 0.006608613301068544
Epoch: 9980 Loss: 0.006607453338801861
Epoch: 9981 Loss: 0.006606157869100571
Epoch: 9982 Loss: 0.006605706177651882
Epoch: 9983 Loss: 0.006603860761970282
Epoch: 9984 Loss: 0.006602838169783354
Epoch: 9985 Loss: 0.006602330598980188
Epoch: 9986 Loss: 0.006601244676858187
Epoch: 9987 Loss: 0.006600590888410807
Epoch: 9988 Loss: 0.006599425803869963
Epoch: 9989 Loss: 0.0065981619991362095
Epoch: 9990 Loss: 0.00659700483083725
Epoch: 9991 Loss: 0.0065963477827608585
Epoch: 9992 Loss: 0.0065948013216257095
Epoch: 9993 Loss: 0.006593888625502586
Epoch: 9994 Loss: 0.006593203637748957
Epoch: 9995 Loss: 0.006592920050024986
Epoch: 9996 Loss: 0.006591069977730513
Epoch: 9997 Loss: 0.006590310949832201
Epoch: 9998 Loss: 0.0065892646089196205
Epoch: 9999 Loss: 0.006588027812540531

最后我们用训练好的模型尝试在1到100这些数字上玩FizzBuzz游戏

# Output now
testX = torch.Tensor([binary_encode(i, NUM_DIGITS) for i in range(1, 101)])
with torch.no_grad():
    testY = model(testX)
predictions = zip(range(1, 101), list(testY.max(1)[1].data.tolist()))

print([fizz_buzz_decode(i, x) for (i, x) in predictions])

['1', '2', 'fizz', '4', 'buzz', 'fizz', '7', '8', 'fizz', 'buzz', '11', 'fizz', '13', '14', 'fizzbuzz', '16', '17', 'fizz', '19', 'buzz', 'fizz', '22', '23', 'fizz', '25', '26', 'fizz', '28', '29', 'fizzbuzz', '31', '32', 'fizz', '34', 'buzz', 'fizz', '37', '38', 'fizz', 'buzz', '41', 'fizz', '43', '44', 'fizzbuzz', '46', '47', 'fizz', '49', 'buzz', 'fizz', '52', '53', 'fizz', 'buzz', '56', 'fizz', '58', '59', 'fizzbuzz', '61', '62', 'fizz', '64', 'buzz', 'fizz', '67', 'buzz', '69', 'buzz', '71', 'fizz', '73', '74', 'fizzbuzz', '76', '77', 'fizz', '79', 'buzz', 'fizz', '82', 'buzz', 'fizz', 'buzz', '86', 'fizz', '88', '89', 'fizzbuzz', '91', '92', 'fizz', '94', 'buzz', 'fizz', '97', '98', 'fizz', 'buzz']

1 2	print(np.sum(testY.max(1)[1].numpy() == np.array([fizz_buzz_encode(i) for i in range(1,101)]))) testY.max(1)[1].numpy() == np.array([fizz_buzz_encode(i) for i in range(1,101)])

96





array([ True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True, False,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True, False, False,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True, False,  True,  True,  True,  True,  True,  True,  True,
        True,  True,  True,  True,  True,  True,  True,  True,  True,
        True])

7.seq2seq

2020-02-29

第六课 Seq2Seq, Attention

褚则伟 zeweichu@gmail.com

在这份notebook当中，我们会(尽可能)复现Luong的attention模型

由于我们的数据集非常小，只有一万多个句子的训练数据，所以训练出来的模型效果并不好。如果大家想训练一个好一点的模型，可以参考下面的资料。

情感分析

褚则伟 zeweichu@gmail.com

学习目标

学习和训练文本分类模型
学习torchtext的基本使用方法
- BucketIterator
学习torch.nn的一些基本模型
- Conv2d

本notebook参考了https://github.com/bentrevett/pytorch-sentiment-analysis

在这份notebook中，我们会用PyTorch模型和TorchText再来做情感分析(检测一段文字的情感是正面的还是负面的)。我们会使用IMDb 数据集，即电影评论。

模型从简单到复杂，我们会依次构建：

Word Averaging模型
RNN/LSTM模型
CNN模型

准备数据

TorchText中的一个重要概念是Field。Field决定了你的数据会被怎样处理。在我们的情感分类任务中，我们所需要接触到的数据有文本字符串和两种情感，”pos”或者”neg”。
Field的参数制定了数据会被怎样处理。
我们使用TEXT field来定义如何处理电影评论，使用LABEL field来处理两个情感类别。
我们的TEXT field带有tokenize='spacy'，这表示我们会用spaCy tokenizer来tokenize英文句子。如果我们不特别声明tokenize这个参数，那么默认的分词方法是使用空格。

安装spaCy

1 2	pip install -U spacy python -m spacy download en

LABEL由LabelField定义。这是一种特别的用来处理label的Field。我们后面会解释dtype。
更多关于Fields，参见https://github.com/pytorch/text/blob/master/torchtext/data/field.py
和之前一样，我们会设定random seeds使实验可以复现。

import torch
import torchtext
from torchtext import data

SEED = 1234

torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TEXT = data.Field(tokenize='spacy')
LABEL = data.LabelField()# dtype=torch.float)

1 2	print(torch.__version__) print(torchtext.__version__)

1.1.0
0.3.1

TorchText支持很多常见的自然语言处理数据集。
下面的代码会自动下载IMDb数据集，然后分成train/test两个torchtext.datasets类别。数据被前面的Fields处理。IMDb数据集一共有50000电影评论，每个评论都被标注为正面的或负面的。

1 2	from torchtext import datasets train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)

查看每个数据split有多少条数据。

1 2	print(f'Number of training examples: {len(train_data)}') print(f'Number of testing examples: {len(test_data)}')

Number of training examples: 25000
Number of testing examples: 25000

查看一个example。

1	print(vars(train_data.examples[0]))

{'text': ['Brilliant', 'adaptation', 'of', 'the', 'novel', 'that', 'made', 'famous', 'the', 'relatives', 'of', 'Chilean', 'President', 'Salvador', 'Allende', 'killed', '.', 'In', 'the', 'environment', 'of', 'a', 'large', 'estate', 'that', 'arises', 'from', 'the', 'ruins', ',', 'becoming', 'a', 'force', 'to', 'abuse', 'and', 'exploitation', 'of', 'outrage', ',', 'a', 'luxury', 'estate', 'for', 'the', 'benefit', 'of', 'the', 'upstart', 'Esteban', 'Trueba', 'and', 'his', 'undeserved', 'family', ',', 'the', 'brilliant', 'Danish', 'director', 'Bille', 'August', 'recreates', ',', 'in', 'micro', ',', 'which', 'at', 'the', 'time', 'would', 'be', 'the', 'process', 'leading', 'to', 'the', 'greatest', 'infamy', 'of', 'his', 'story', 'to', 'the', 'hardened', 'Chilean', 'nation', ',', 'and', 'whose', 'main', 'character', 'would', 'Augusto', 'Pinochet', '(', 'Stephen', 'similarities', 'with', 'it', 'are', 'inevitable', ':', 'recall', ',', 'as', 'an', 'example', ',', 'that', 'image', 'of', 'the', 'senator', 'with', 'dark', 'glasses', 'that', 'makes', 'him', 'the', 'wink', 'to', 'the', 'general', 'to', 'begin', 'making', 'the', 'palace).<br', '/><br', '/>Bille', 'August', 'attends', 'an', 'exceptional', 'cast', 'in', 'the', 'Jeremy', 'protruding', 'Irons', ',', 'whose', 'character', 'changes', 'from', 'arrogance', 'and', 'extreme', 'cruelty', ',', 'the', 'hard', 'lesson', 'that', 'life', 'always', 'brings', 'us', 'to', 'almost', 'force', 'us', 'to', 'change', '.', 'In', 'Esteban', 'fully', 'applies', 'the', 'law', 'of', 'resonance', ',', 'with', 'great', 'wisdom', ',', 'Solomon', 'describes', 'in', 'these', 'words:"The', 'things', 'that', 'freckles', 'are', 'the', 'same', 'punishment', 'that', 'will', 'serve', 'you', '.', '"', '<', 'br', '/><br', '/>Unforgettable', 'Glenn', 'Close', 'playing', 'splint', ',', 'the', 'tainted', 'sister', 'of', 'Stephen', ',', 'whose', 'sin', ',', 'driven', 'by', 'loneliness', ',', 'spiritual', 'and', 'platonic', 'love', 'was', 'the', 'wife', 'of', 'his', 'cruel', 'snowy', 'brother', '.', 'Meryl', 'Streep', 'also', 'brilliant', ',', 'a', 'woman', 'whose', 'name', 'came', 'to', 'him', 'like', 'a', 'glove', 'Clara', '.', 'With', 'telekinetic', 'powers', ',', 'cognitive', 'and', 'mediumistic', ',', 'this', 'hardened', 'woman', ',', 'loyal', 'to', 'his', 'blunt', ',', 'conservative', 'husband', ',', 'is', 'an', 'indicator', 'of', 'character', 'and', 'self', '-', 'control', 'that', 'we', 'wish', 'for', 'ourselves', 'and', 'for', 'all', 'human', 'beings', '.', '<', 'br', '/><br', '/>Every', 'character', 'is', 'a', 'portrait', 'of', 'virtuosity', '(', 'as', 'Blanca', 'worthy', 'rebel', 'leader', 'Pedro', 'Segundo', 'unhappy', '...', ')', 'or', 'a', 'portrait', 'of', 'humiliation', ',', 'like', 'Stephen', 'Jr.', ',', 'the', 'bastard', 'child', 'of', 'Senator', ',', 'who', 'serves', 'as', 'an', 'instrument', 'for', 'the', 'return', 'of', 'the', 'boomerang', '.', '<', 'br', '/><br', '/>The', 'film', 'moves', 'the', 'bowels', ',', 'we', 'recreated', 'some', 'facts', 'that', 'should', 'not', 'ever', 'be', 'repeated', ',', 'but', 'that', 'absurdly', 'still', 'happen', '(', 'Colombia', 'is', 'a', 'sad', 'example', ')', 'and', 'another', 'reminder', 'that', ',', 'against', 'all', ',', 'life', 'is', 'wonderful', 'because', 'there', 'are', 'always', 'people', 'like', 'Isabel', 'Allende', 'and', 'immortalize', 'just', 'Bille', 'August', '.'], 'label': 'pos'}

由于我们现在只有train/test这两个分类，所以我们需要创建一个新的validation set。我们可以使用.split()创建新的分类。
默认的数据分割是 70、30，如果我们声明split_ratio，可以改变split之间的比例，split_ratio=0.8表示80%的数据是训练集，20%是验证集。
我们还声明random_state这个参数，确保我们每次分割的数据集都是一样的。

1 2	import random train_data, valid_data = train_data.split(random_state=random.seed(SEED))

检查一下现在每个部分有多少条数据。

1
2
3

print(f'Number of training examples: {len(train_data)}')
print(f'Number of validation examples: {len(valid_data)}')
print(f'Number of testing examples: {len(test_data)}')

Number of training examples: 17500
Number of validation examples: 7500
Number of testing examples: 25000

下一步我们需要创建 vocabulary 。vocabulary 就是把每个单词一一映射到一个数字。
我们使用最常见的25k个单词来构建我们的单词表，用max_size这个参数可以做到这一点。
所有其他的单词都用<unk>来表示。

# TEXT.build_vocab(train_data, max_size=25000)
# LABEL.build_vocab(train_data)
TEXT.build_vocab(train_data, max_size=25000, vectors="glove.6B.100d", unk_init=torch.Tensor.normal_)
LABEL.build_vocab(train_data)

1 2	print(f"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}") print(f"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}")

Unique tokens in TEXT vocabulary: 25002
Unique tokens in LABEL vocabulary: 2

当我们把句子传进模型的时候，我们是按照一个个 batch 穿进去的，也就是说，我们一次传入了好几个句子，而且每个batch中的句子必须是相同的长度。为了确保句子的长度相同，TorchText会把短的句子pad到和最长的句子等长。
下面我们来看看训练数据集中最常见的单词。

1	print(TEXT.vocab.freqs.most_common(20))

[('the', 201455), (',', 192552), ('.', 164402), ('a', 108963), ('and', 108649), ('of', 100010), ('to', 92873), ('is', 76046), ('in', 60904), ('I', 54486), ('it', 53405), ('that', 49155), ('"', 43890), ("'s", 43151), ('this', 42454), ('-', 36769), ('/><br', 35511), ('was', 34990), ('as', 30324), ('with', 29691)]

我们可以直接用 stoi(string to int) 或者 itos (int to string) 来查看我们的单词表。

1	print(TEXT.vocab.itos[:10])

['<unk>', '<pad>', 'the', ',', '.', 'a', 'and', 'of', 'to', 'is']

查看labels。

1	print(LABEL.vocab.stoi)

defaultdict(<function _default_unk_index at 0x7f6944d3f730>, {'neg': 0, 'pos': 1})

最后一步数据的准备是创建iterators。每个itartion都会返回一个batch的examples。
我们会使用BucketIterator。BucketIterator会把长度差不多的句子放到同一个batch中，确保每个batch中不出现太多的padding。
严格来说，我们这份notebook中的模型代码都有一个问题，也就是我们把<pad>也当做了模型的输入进行训练。更好的做法是在模型中把由<pad>产生的输出给消除掉。在这节课中我们简单处理，直接把<pad>也用作模型输入了。由于<pad>数量不多，模型的效果也不差。
如果我们有GPU，还可以指定每个iteration返回的tensor都在GPU上。

BATCH_SIZE = 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data), 
    batch_size=BATCH_SIZE,
    device=device,
    repeat=False)

1 2	# for i, _ in enumerate(train_iterator): # print(i)

1	batch = next(iter(train_iterator))

1	batch.text

tensor([[   65,  6706,    23,  ...,  3101,    54,    87],
        [   52, 11017,    83,  ..., 24113,    15,  1078],
        [    8,     3,   671,  ...,    52,    73,     3],
        ...,
        [    1,     1,     1,  ...,     1,     1,     1],
        [    1,     1,     1,  ...,     1,     1,     1],
        [    1,     1,     1,  ...,     1,     1,     1]], device='cuda:0')

1
2
3

PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]
mask = batch.text == PAD_IDX
mask

tensor([[0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        [0, 0, 0,  ..., 0, 0, 0],
        ...,
        [1, 1, 1,  ..., 1, 1, 1],
        [1, 1, 1,  ..., 1, 1, 1],
        [1, 1, 1,  ..., 1, 1, 1]], device='cuda:0', dtype=torch.uint8)

Word Averaging模型

我们首先介绍一个简单的Word Averaging模型。这个模型非常简单，我们把每个单词都通过Embedding层投射成word embedding vector，然后把一句话中的所有word vector做个平均，就是整个句子的vector表示了。接下来把这个sentence vector传入一个Linear层，做分类即可。

我们使用avg_pool2d来做average pooling。我们的目标是把sentence length那个维度平均成1，然后保留embedding这个维度。

avg_pool2d的kernel size是 (embedded.shape[1], 1)，所以句子长度的那个维度会被压扁。

import torch.nn as nn
import torch.nn.functional as F

class WordAVGModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, output_dim, pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.fc = nn.Linear(embedding_dim, output_dim)
        
    def forward(self, text, mask):
        embedded = self.embedding(text) # [batch_size, seq_len, emb_dim]
        sent_embed = torch.sum(embedded * mask.unsqueeze(2), 1) / mask.sum(1).unsqueeze(1) # [batch size, embedding_dim]
        return self.fc(sent_embed)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
OUTPUT_DIM = 1

model = WordAVGModel(INPUT_DIM, EMBEDDING_DIM, OUTPUT_DIM, PAD_IDX)

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 2,500,301 trainable parameters

1 2	pretrained_embeddings = TEXT.vocab.vectors model.embedding.weight.data.copy_(pretrained_embeddings)

tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],
        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])

UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

训练模型

import torch.optim as optim

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)

计算预测的准确率

def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    #round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float() #convert into float for division 
    acc = correct.sum()/len(correct)
    return acc

def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    
    i = 0
    for batch in iterator:
        optimizer.zero_grad()
        # CHANGED
        text = batch.text.permute(1, 0) # [batch_size, seq_length]
        mask = 1. - (text == PAD_IDX).float() # [batch_size, seq_len]
        predictions = model(text, mask).squeeze(1)
        loss = criterion(predictions, batch.label.float())
        acc = binary_accuracy(predictions, batch.label.float())
        loss.backward()
        optimizer.step()
        
        if i % 100 == 0:
            print("batch {}, loss {}".format(i, loss.item()))
        i += 1
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.eval()
    
    i = 0
    with torch.no_grad():
        for batch in iterator:
            text = batch.text.permute(1, 0) # [batch_size, seq_length]
            mask = 1. - (text == PAD_IDX).float() # [batch_size, seq_len]
            predictions = model(text, mask).squeeze(1)
            loss = criterion(predictions, batch.label.float())
            acc = binary_accuracy(predictions, batch.label.float())
            
            if i % 100 == 0:
                print("batch {}, loss {}".format(i, loss.item()))
            i += 1
            epoch_loss += loss.item()
            epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

N_EPOCHS = 10

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'wordavg-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

batch 0, loss 0.6835149526596069
batch 100, loss 0.6759217977523804
batch 200, loss 0.6536192297935486
batch 0, loss 0.5802608132362366
batch 100, loss 0.6405552625656128
Epoch: 01 | Epoch Time: 0m 2s
    Train Loss: 0.661 | Train Acc: 66.62%
     Val. Loss: 0.615 |  Val. Acc: 74.25%
batch 0, loss 0.6175215244293213
batch 100, loss 0.5193076133728027
batch 200, loss 0.523094654083252
batch 0, loss 0.41260701417922974
batch 100, loss 0.546144425868988
Epoch: 02 | Epoch Time: 0m 2s
    Train Loss: 0.542 | Train Acc: 78.82%
     Val. Loss: 0.482 |  Val. Acc: 81.45%
batch 0, loss 0.48719578981399536
batch 100, loss 0.3965785503387451
batch 200, loss 0.4322021007537842
batch 0, loss 0.35118478536605835
batch 100, loss 0.46531984210014343
Epoch: 03 | Epoch Time: 0m 2s
    Train Loss: 0.414 | Train Acc: 85.14%
     Val. Loss: 0.391 |  Val. Acc: 85.33%
batch 0, loss 0.31555071473121643
batch 100, loss 0.3576723039150238
batch 200, loss 0.43358099460601807
batch 0, loss 0.3284790515899658
batch 100, loss 0.4068619906902313
Epoch: 04 | Epoch Time: 0m 2s
    Train Loss: 0.333 | Train Acc: 88.22%
     Val. Loss: 0.341 |  Val. Acc: 86.73%
batch 0, loss 0.21446196734905243
batch 100, loss 0.29952651262283325
batch 200, loss 0.33016496896743774
batch 0, loss 0.33019396662712097
batch 100, loss 0.372672975063324
Epoch: 05 | Epoch Time: 0m 2s
    Train Loss: 0.284 | Train Acc: 90.08%
     Val. Loss: 0.311 |  Val. Acc: 87.85%
batch 0, loss 0.21933476626873016
batch 100, loss 0.20656771957874298
batch 200, loss 0.2411007285118103
batch 0, loss 0.3338389992713928
batch 100, loss 0.35051852464675903
Epoch: 06 | Epoch Time: 0m 2s
    Train Loss: 0.248 | Train Acc: 91.57%
     Val. Loss: 0.292 |  Val. Acc: 88.37%
batch 0, loss 0.2381495237350464
batch 100, loss 0.3066502809524536
batch 200, loss 0.17593657970428467
batch 0, loss 0.33260178565979004
batch 100, loss 0.3287006616592407
Epoch: 07 | Epoch Time: 0m 2s
    Train Loss: 0.220 | Train Acc: 92.62%
     Val. Loss: 0.281 |  Val. Acc: 88.89%
batch 0, loss 0.18733319640159607
batch 100, loss 0.2353360801935196
batch 200, loss 0.19918608665466309
batch 0, loss 0.34648358821868896
batch 100, loss 0.3191569447517395
Epoch: 08 | Epoch Time: 0m 2s
    Train Loss: 0.197 | Train Acc: 93.63%
     Val. Loss: 0.269 |  Val. Acc: 89.23%
batch 0, loss 0.10634639114141464
batch 100, loss 0.11403544247150421
batch 200, loss 0.29342859983444214
batch 0, loss 0.35649430751800537
batch 100, loss 0.3183209300041199
Epoch: 09 | Epoch Time: 0m 2s
    Train Loss: 0.177 | Train Acc: 94.27%
     Val. Loss: 0.264 |  Val. Acc: 89.26%
batch 0, loss 0.16292411088943481
batch 100, loss 0.08687698841094971
batch 200, loss 0.21162091195583344
batch 0, loss 0.3467680811882019
batch 100, loss 0.2997514605522156
Epoch: 10 | Epoch Time: 0m 2s
    Train Loss: 0.160 | Train Acc: 94.98%
     Val. Loss: 0.258 |  Val. Acc: 89.72%

import spacy
nlp = spacy.load('en')

def predict_sentiment(sentence):
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    tensor = torch.LongTensor(indexed).to(device)
    text = tensor.unsqueeze(0)
    mask = 1. - (text == PAD_IDX).float() # [batch_size, seq_len]
    prediction = torch.sigmoid(model(tensor, mask))
    return prediction.item()

1	predict_sentiment("This film is terrible")

2.4536811471520537e-10

1	predict_sentiment("This film is great")

1.0

1
2
3

model.load_state_dict(torch.load('wordavg-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

batch 0, loss 0.32167336344718933
batch 100, loss 0.34431976079940796
batch 200, loss 0.18615691363811493
batch 300, loss 0.37860944867134094
Test Loss: 0.290 | Test Acc: 88.03%

RNN模型

下面我们尝试把模型换成一个recurrent neural network (RNN)。RNN经常会被用来encode一个sequence
$$h_t = \text{RNN}(x_t, h_{t-1})$$
我们使用最后一个hidden state $h_T$来表示整个句子。
然后我们把$h_T$通过一个线性变换$f$，然后用来预测句子的情感。

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, 
                 n_layers, bidirectional, dropout, pad_idx, avg_hidden=True):
        super().__init__()
        self.bidirectional = bidirectional
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers=n_layers, 
                           bidirectional=bidirectional, dropout=dropout)
        
        self.fc = nn.Linear(hidden_dim*2 if self.bidirectional else hidden_dim, output_dim)
        self.dropout = nn.Dropout(dropout)
        self.avg_hidden = avg_hidden
        
    def forward(self, text, mask):
        embedded = self.dropout(self.embedding(text)) #[sent len, batch size, emb dim]
        # CHANGED
        seq_length = mask.sum(1)
        embedded = torch.nn.utils.rnn.pack_padded_sequence(
            input=embedded, 
            lengths=seq_length, 
            batch_first=True,
            enforce_sorted=False)
        output, (hidden, cell) = self.rnn(embedded)
        output, seq_length = torch.nn.utils.rnn.pad_packed_sequence(
            sequence=output,
            batch_first=True,
            padding_value=0,
            total_length=mask.shape[1]
        )
        #output = [sent len, batch size, hid dim * num directions]
        #hidden = [num layers * num directions, batch size, hid dim]
        #cell = [num layers * num directions, batch size, hid dim]
        
        if self.avg_hidden:
#             print(output)
            hidden = torch.sum(output * mask.unsqueeze(2), 1) / torch.sum(mask, 1, keepdim=True) # [batch size, embedding_dim]
        else:
            if self.bidirectional:
                #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
                hidden = torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1) # [batch size, hid dim * num directions]
            else:
                hidden = self.dropout(hidden[-1,:,:]) # [batch size, hid dim * num directions]
        # apply dropout
        hidden = self.dropout(hidden)
        return self.fc(hidden)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, 
            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX, avg_hidden=False)

1	print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 4,810,857 trainable parameters

model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

print(model.embedding.weight.data)

tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])

训练RNN模型

1 2	optimizer = optim.Adam(model.parameters()) model = model.to(device)

N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'lstm-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

batch 0, loss 0.6940298080444336
batch 100, loss 0.6605077981948853
batch 200, loss 0.5677657723426819
batch 0, loss 0.6464325189590454
batch 100, loss 0.7902224659919739
Epoch: 01 | Epoch Time: 1m 1s
    Train Loss: 0.651 | Train Acc: 61.65%
     Val. Loss: 0.717 |  Val. Acc: 52.98%
batch 0, loss 0.7926035523414612
batch 100, loss 0.7492727637290955
batch 200, loss 0.7025203704833984
batch 0, loss 0.6599957942962646
batch 100, loss 0.6523773670196533
Epoch: 02 | Epoch Time: 1m 1s
    Train Loss: 0.673 | Train Acc: 57.12%
     Val. Loss: 0.659 |  Val. Acc: 61.20%
batch 0, loss 0.64130699634552
batch 100, loss 0.6027564406394958
batch 200, loss 0.6683254837989807
batch 0, loss 0.5396684408187866
batch 100, loss 0.5652653574943542
Epoch: 03 | Epoch Time: 1m 2s
    Train Loss: 0.610 | Train Acc: 66.25%
     Val. Loss: 0.597 |  Val. Acc: 68.90%
batch 0, loss 0.580141544342041
batch 100, loss 0.2638660669326782
batch 200, loss 0.4949319064617157
batch 0, loss 0.3330756723880768
batch 100, loss 0.39001500606536865
Epoch: 04 | Epoch Time: 1m 1s
    Train Loss: 0.479 | Train Acc: 77.27%
     Val. Loss: 0.378 |  Val. Acc: 84.53%
batch 0, loss 0.4124695062637329
batch 100, loss 0.5047512054443359
batch 200, loss 0.4246818423271179
batch 0, loss 0.3377535343170166
batch 100, loss 0.29955512285232544
Epoch: 05 | Epoch Time: 1m 1s
    Train Loss: 0.343 | Train Acc: 85.52%
     Val. Loss: 0.309 |  Val. Acc: 87.28%

我们来尝试一个AVG的版本

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

rnn_model_avg = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, 
            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)

1	print(f'The model has {count_parameters(rnn_model_avg):,} trainable parameters')

The model has 4,810,857 trainable parameters

rnn_model_avg.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

rnn_model_avg.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
rnn_model_avg.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

print(rnn_model_avg.embedding.weight.data)

tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])

训练RNN模型

1 2	optimizer = optim.Adam(rnn_model_avg.parameters()) rnn_model_avg = rnn_model_avg.to(device)

N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(rnn_model_avg, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(rnn_model_avg, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(rnn_model_avg.state_dict(), 'lstm-avg-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

batch 0, loss 0.6885155439376831
batch 100, loss 0.5888913869857788
batch 200, loss 0.4656108617782593
batch 0, loss 0.4603933095932007
batch 100, loss 0.38754644989967346
Epoch: 01 | Epoch Time: 1m 20s
    Train Loss: 0.528 | Train Acc: 72.70%
     Val. Loss: 0.362 |  Val. Acc: 84.47%
batch 0, loss 0.29848513007164
batch 100, loss 0.27336984872817993
batch 200, loss 0.35852643847465515
batch 0, loss 0.4745270907878876
batch 100, loss 0.32764753699302673
Epoch: 02 | Epoch Time: 1m 20s
    Train Loss: 0.342 | Train Acc: 85.55%
     Val. Loss: 0.294 |  Val. Acc: 88.03%
batch 0, loss 0.31138738989830017
batch 100, loss 0.3301498591899872
batch 200, loss 0.5036394596099854
batch 0, loss 0.36463940143585205
batch 100, loss 0.3079427480697632
Epoch: 03 | Epoch Time: 1m 20s
    Train Loss: 0.276 | Train Acc: 88.91%
     Val. Loss: 0.257 |  Val. Acc: 89.85%
batch 0, loss 0.19154249131679535
batch 100, loss 0.24453845620155334
batch 200, loss 0.2616804540157318
batch 0, loss 0.4100673198699951
batch 100, loss 0.29790183901786804
Epoch: 04 | Epoch Time: 1m 20s
    Train Loss: 0.230 | Train Acc: 91.25%
     Val. Loss: 0.250 |  Val. Acc: 90.16%
batch 0, loss 0.21265330910682678
batch 100, loss 0.34193551540374756
batch 200, loss 0.19812607765197754
batch 0, loss 0.3696991205215454
batch 100, loss 0.30417782068252563
Epoch: 05 | Epoch Time: 1m 20s
    Train Loss: 0.202 | Train Acc: 92.49%
     Val. Loss: 0.251 |  Val. Acc: 90.26%

You may have noticed the loss is not really decreasing and the accuracy is poor. This is due to several issues with the model which we’ll improve in the next notebook.

Finally, the metric we actually care about, the test loss and accuracy, which we get from our parameters that gave us the best validation loss.

1
2
3

rnn_model_avg.load_state_dict(torch.load('lstm-avg-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

batch 0, loss 0.24243609607219696
batch 100, loss 0.28235137462615967
batch 200, loss 0.20145781338214874
batch 300, loss 0.3198160231113434
Test Loss: 0.335 | Test Acc: 85.88%

CNN模型

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, 
                 filter_sizes, output_dim, dropout, pad_idx):
        super().__init__()
        
        self.filter_sizes = filter_sizes
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text, mask):
        # CHANGED
        embedded = self.embedding(text) # [batch size, sent len, emb dim]
        embedded = embedded.unsqueeze(1) # [batch size, 1, sent len, emb dim]
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]
        
#         print((1.-mask[:, :-3+1]).unsqueeze(1).byte().shape)
        conved = [conv.masked_fill((1.-mask[:, :-filter_size+1]).unsqueeze(1).byte(), -999999) \
                  for (conv, filter_size) in zip(conved, self.filter_sizes)]
        
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        
        #pooled_n = [batch size, n_filters]
        
        cat = self.dropout(torch.cat(pooled, dim=1))

        #cat = [batch size, n_filters * len(filter_sizes)]
            
        return self.fc(cat)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
N_FILTERS = 100
FILTER_SIZES = [3,4,5]
OUTPUT_DIM = 1
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]


model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)
model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)
model = model.to(device)

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
criterion = criterion.to(device)

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'CNN-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

batch 0, loss 0.7456250190734863
batch 100, loss 0.7356712818145752
batch 200, loss 0.608451247215271
batch 0, loss 0.5171981453895569
batch 100, loss 0.5627424716949463
Epoch: 01 | Epoch Time: 0m 11s
    Train Loss: 0.653 | Train Acc: 61.19%
     Val. Loss: 0.511 |  Val. Acc: 78.05%
batch 0, loss 0.5206002593040466
batch 100, loss 0.4522325098514557
batch 200, loss 0.39397668838500977
batch 0, loss 0.36625632643699646
batch 100, loss 0.34350645542144775
Epoch: 02 | Epoch Time: 0m 11s
    Train Loss: 0.430 | Train Acc: 80.41%
     Val. Loss: 0.356 |  Val. Acc: 85.21%
batch 0, loss 0.3453408479690552
batch 100, loss 0.3106832504272461
batch 200, loss 0.29214251041412354
batch 0, loss 0.34314772486686707
batch 100, loss 0.27926790714263916
Epoch: 03 | Epoch Time: 0m 11s
    Train Loss: 0.305 | Train Acc: 87.17%
     Val. Loss: 0.318 |  Val. Acc: 86.52%
batch 0, loss 0.2820616066455841
batch 100, loss 0.2185526192188263
batch 200, loss 0.2295588254928589
batch 0, loss 0.3212977647781372
batch 100, loss 0.2501620352268219
Epoch: 04 | Epoch Time: 0m 11s
    Train Loss: 0.222 | Train Acc: 91.30%
     Val. Loss: 0.311 |  Val. Acc: 87.25%
batch 0, loss 0.06584674119949341
batch 100, loss 0.1338910311460495
batch 200, loss 0.22213703393936157
batch 0, loss 0.32934656739234924
batch 100, loss 0.2596980333328247
Epoch: 05 | Epoch Time: 0m 11s
    Train Loss: 0.162 | Train Acc: 94.01%
     Val. Loss: 0.318 |  Val. Acc: 87.29%

1
2
3

model.load_state_dict(torch.load('CNN-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

batch 0, loss 0.1641087532043457
batch 100, loss 0.38564836978912354
batch 200, loss 0.26448047161102295
batch 300, loss 0.4913085401058197
Test Loss: 0.350 | Test Acc: 85.04%

4.sentiment

2020-02-29

第四课情感分析

褚则伟 zeweichu@gmail.com

学习目标

学习和训练文本分类模型
学习torchtext的基本使用方法
- BucketIterator
学习torch.nn的一些基本模型
- Conv2d

本notebook参考了https://github.com/bentrevett/pytorch-sentiment-analysis

在这份notebook中，我们会用PyTorch模型和TorchText再来做情感分析(检测一段文字的情感是正面的还是负面的)。我们会使用IMDb 数据集，即电影评论。

模型从简单到复杂，我们会依次构建：

Word Averaging模型
RNN/LSTM模型
CNN模型

准备数据

TorchText中的一个重要概念是Field。Field决定了你的数据会被怎样处理。在我们的情感分类任务中，我们所需要接触到的数据有文本字符串和两种情感，”pos”或者”neg”。
Field的参数制定了数据会被怎样处理。
我们使用TEXT field来定义如何处理电影评论，使用LABEL field来处理两个情感类别。
我们的TEXT field带有tokenize='spacy'，这表示我们会用spaCy tokenizer来tokenize英文句子。如果我们不特别声明tokenize这个参数，那么默认的分词方法是使用空格。

安装spaCy

1 2	pip install -U spacy python -m spacy download en

LABEL由LabelField定义。这是一种特别的用来处理label的Field。我们后面会解释dtype。
更多关于Fields，参见https://github.com/pytorch/text/blob/master/torchtext/data/field.py
和之前一样，我们会设定random seeds使实验可以复现。

import torch
from torchtext import data

SEED = 1234

torch.manual_seed(SEED)
torch.cuda.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

TEXT = data.Field(tokenize='spacy')
LABEL = data.LabelField(dtype=torch.float)

TorchText支持很多常见的自然语言处理数据集。
下面的代码会自动下载IMDb数据集，然后分成train/test两个torchtext.datasets类别。数据被前面的Fields处理。IMDb数据集一共有50000电影评论，每个评论都被标注为正面的或负面的。

1 2	from torchtext import datasets train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)

查看每个数据split有多少条数据。

1 2	print(f'Number of training examples: {len(train_data)}') print(f'Number of testing examples: {len(test_data)}')

Number of training examples: 25000
Number of testing examples: 25000

查看一个example。

1	print(vars(train_data.examples[0]))

{'text': ['Brilliant', 'adaptation', 'of', 'the', 'novel', 'that', 'made', 'famous', 'the', 'relatives', 'of', 'Chilean', 'President', 'Salvador', 'Allende', 'killed', '.', 'In', 'the', 'environment', 'of', 'a', 'large', 'estate', 'that', 'arises', 'from', 'the', 'ruins', ',', 'becoming', 'a', 'force', 'to', 'abuse', 'and', 'exploitation', 'of', 'outrage', ',', 'a', 'luxury', 'estate', 'for', 'the', 'benefit', 'of', 'the', 'upstart', 'Esteban', 'Trueba', 'and', 'his', 'undeserved', 'family', ',', 'the', 'brilliant', 'Danish', 'director', 'Bille', 'August', 'recreates', ',', 'in', 'micro', ',', 'which', 'at', 'the', 'time', 'would', 'be', 'the', 'process', 'leading', 'to', 'the', 'greatest', 'infamy', 'of', 'his', 'story', 'to', 'the', 'hardened', 'Chilean', 'nation', ',', 'and', 'whose', 'main', 'character', 'would', 'Augusto', 'Pinochet', '(', 'Stephen', 'similarities', 'with', 'it', 'are', 'inevitable', ':', 'recall', ',', 'as', 'an', 'example', ',', 'that', 'image', 'of', 'the', 'senator', 'with', 'dark', 'glasses', 'that', 'makes', 'him', 'the', 'wink', 'to', 'the', 'general', 'to', 'begin', 'making', 'the', 'palace).<br', '/><br', '/>Bille', 'August', 'attends', 'an', 'exceptional', 'cast', 'in', 'the', 'Jeremy', 'protruding', 'Irons', ',', 'whose', 'character', 'changes', 'from', 'arrogance', 'and', 'extreme', 'cruelty', ',', 'the', 'hard', 'lesson', 'that', 'life', 'always', 'brings', 'us', 'to', 'almost', 'force', 'us', 'to', 'change', '.', 'In', 'Esteban', 'fully', 'applies', 'the', 'law', 'of', 'resonance', ',', 'with', 'great', 'wisdom', ',', 'Solomon', 'describes', 'in', 'these', 'words:"The', 'things', 'that', 'freckles', 'are', 'the', 'same', 'punishment', 'that', 'will', 'serve', 'you', '.', '"', '<', 'br', '/><br', '/>Unforgettable', 'Glenn', 'Close', 'playing', 'splint', ',', 'the', 'tainted', 'sister', 'of', 'Stephen', ',', 'whose', 'sin', ',', 'driven', 'by', 'loneliness', ',', 'spiritual', 'and', 'platonic', 'love', 'was', 'the', 'wife', 'of', 'his', 'cruel', 'snowy', 'brother', '.', 'Meryl', 'Streep', 'also', 'brilliant', ',', 'a', 'woman', 'whose', 'name', 'came', 'to', 'him', 'like', 'a', 'glove', 'Clara', '.', 'With', 'telekinetic', 'powers', ',', 'cognitive', 'and', 'mediumistic', ',', 'this', 'hardened', 'woman', ',', 'loyal', 'to', 'his', 'blunt', ',', 'conservative', 'husband', ',', 'is', 'an', 'indicator', 'of', 'character', 'and', 'self', '-', 'control', 'that', 'we', 'wish', 'for', 'ourselves', 'and', 'for', 'all', 'human', 'beings', '.', '<', 'br', '/><br', '/>Every', 'character', 'is', 'a', 'portrait', 'of', 'virtuosity', '(', 'as', 'Blanca', 'worthy', 'rebel', 'leader', 'Pedro', 'Segundo', 'unhappy', '...', ')', 'or', 'a', 'portrait', 'of', 'humiliation', ',', 'like', 'Stephen', 'Jr.', ',', 'the', 'bastard', 'child', 'of', 'Senator', ',', 'who', 'serves', 'as', 'an', 'instrument', 'for', 'the', 'return', 'of', 'the', 'boomerang', '.', '<', 'br', '/><br', '/>The', 'film', 'moves', 'the', 'bowels', ',', 'we', 'recreated', 'some', 'facts', 'that', 'should', 'not', 'ever', 'be', 'repeated', ',', 'but', 'that', 'absurdly', 'still', 'happen', '(', 'Colombia', 'is', 'a', 'sad', 'example', ')', 'and', 'another', 'reminder', 'that', ',', 'against', 'all', ',', 'life', 'is', 'wonderful', 'because', 'there', 'are', 'always', 'people', 'like', 'Isabel', 'Allende', 'and', 'immortalize', 'just', 'Bille', 'August', '.'], 'label': 'pos'}

由于我们现在只有train/test这两个分类，所以我们需要创建一个新的validation set。我们可以使用.split()创建新的分类。
默认的数据分割是 70、30，如果我们声明split_ratio，可以改变split之间的比例，split_ratio=0.8表示80%的数据是训练集，20%是验证集。
我们还声明random_state这个参数，确保我们每次分割的数据集都是一样的。

1 2	import random train_data, valid_data = train_data.split(random_state=random.seed(SEED))

检查一下现在每个部分有多少条数据。

1
2
3

print(f'Number of training examples: {len(train_data)}')
print(f'Number of validation examples: {len(valid_data)}')
print(f'Number of testing examples: {len(test_data)}')

Number of training examples: 17500
Number of validation examples: 7500
Number of testing examples: 25000

下一步我们需要创建 vocabulary 。vocabulary 就是把每个单词一一映射到一个数字。
我们使用最常见的25k个单词来构建我们的单词表，用max_size这个参数可以做到这一点。
所有其他的单词都用<unk>来表示。

# TEXT.build_vocab(train_data, max_size=25000)
# LABEL.build_vocab(train_data)
TEXT.build_vocab(train_data, max_size=25000, vectors="glove.6B.100d", unk_init=torch.Tensor.normal_)
LABEL.build_vocab(train_data)

1 2	print(f"Unique tokens in TEXT vocabulary: {len(TEXT.vocab)}") print(f"Unique tokens in LABEL vocabulary: {len(LABEL.vocab)}")

Unique tokens in TEXT vocabulary: 25002
Unique tokens in LABEL vocabulary: 2

当我们把句子传进模型的时候，我们是按照一个个 batch 穿进去的，也就是说，我们一次传入了好几个句子，而且每个batch中的句子必须是相同的长度。为了确保句子的长度相同，TorchText会把短的句子pad到和最长的句子等长。
下面我们来看看训练数据集中最常见的单词。

1	print(TEXT.vocab.freqs.most_common(20))

[('the', 201455), (',', 192552), ('.', 164402), ('a', 108963), ('and', 108649), ('of', 100010), ('to', 92873), ('is', 76046), ('in', 60904), ('I', 54486), ('it', 53405), ('that', 49155), ('"', 43890), ("'s", 43151), ('this', 42454), ('-', 36769), ('/><br', 35511), ('was', 34990), ('as', 30324), ('with', 29691)]

我们可以直接用 stoi(string to int) 或者 itos (int to string) 来查看我们的单词表。

1	print(TEXT.vocab.itos[:10])

['<unk>', '<pad>', 'the', ',', '.', 'a', 'and', 'of', 'to', 'is']

查看labels。

1	print(LABEL.vocab.stoi)

defaultdict(<function _default_unk_index at 0x7fbec39a79d8>, {'neg': 0, 'pos': 1})

最后一步数据的准备是创建iterators。每个itartion都会返回一个batch的examples。
我们会使用BucketIterator。BucketIterator会把长度差不多的句子放到同一个batch中，确保每个batch中不出现太多的padding。
严格来说，我们这份notebook中的模型代码都有一个问题，也就是我们把<pad>也当做了模型的输入进行训练。更好的做法是在模型中把由<pad>产生的输出给消除掉。在这节课中我们简单处理，直接把<pad>也用作模型输入了。由于<pad>数量不多，模型的效果也不差。
如果我们有GPU，还可以指定每个iteration返回的tensor都在GPU上。

BATCH_SIZE = 64

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data), 
    batch_size=BATCH_SIZE,
    device=device)

Word Averaging模型

我们首先介绍一个简单的Word Averaging模型。这个模型非常简单，我们把每个单词都通过Embedding层投射成word embedding vector，然后把一句话中的所有word vector做个平均，就是整个句子的vector表示了。接下来把这个sentence vector传入一个Linear层，做分类即可。

我们使用avg_pool2d来做average pooling。我们的目标是把sentence length那个维度平均成1，然后保留embedding这个维度。

avg_pool2d的kernel size是 (embedded.shape[1], 1)，所以句子长度的那个维度会被压扁。

import torch.nn as nn
import torch.nn.functional as F

class WordAVGModel(nn.Module):
    def __init__(self, vocab_size, embedding_dim, output_dim, pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.fc = nn.Linear(embedding_dim, output_dim)
        
    def forward(self, text):
        embedded = self.embedding(text) # [sent len, batch size, emb dim]
        embedded = embedded.permute(1, 0, 2) # [batch size, sent len, emb dim]
        pooled = F.avg_pool2d(embedded, (embedded.shape[1], 1)).squeeze(1) # [batch size, embedding_dim]
        return self.fc(pooled)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
OUTPUT_DIM = 1
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = WordAVGModel(INPUT_DIM, EMBEDDING_DIM, OUTPUT_DIM, PAD_IDX)

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 2,500,301 trainable parameters

1 2	pretrained_embeddings = TEXT.vocab.vectors model.embedding.weight.data.copy_(pretrained_embeddings)

tensor([[-0.1117, -0.4966,  0.1631,  ...,  1.2647, -0.2753, -0.1325],
        [-0.8555, -0.7208,  1.3755,  ...,  0.0825, -1.1314,  0.3997],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]])

UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

训练模型

import torch.optim as optim

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)

计算预测的准确率

def binary_accuracy(preds, y):
    """
    Returns accuracy per batch, i.e. if you get 8/10 right, this returns 0.8, NOT 8
    """

    #round predictions to the closest integer
    rounded_preds = torch.round(torch.sigmoid(preds))
    correct = (rounded_preds == y).float() #convert into float for division 
    acc = correct.sum()/len(correct)
    return acc

def train(model, iterator, optimizer, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    
    for batch in iterator:
        optimizer.zero_grad()
        predictions = model(batch.text).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = binary_accuracy(predictions, batch.label)
        loss.backward()
        optimizer.step()
        
        epoch_loss += loss.item()
        epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

def evaluate(model, iterator, criterion):
    
    epoch_loss = 0
    epoch_acc = 0
    model.eval()
    
    with torch.no_grad():
        for batch in iterator:
            predictions = model(batch.text).squeeze(1)
            loss = criterion(predictions, batch.label)
            acc = binary_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
        
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

import time

def epoch_time(start_time, end_time):
    elapsed_time = end_time - start_time
    elapsed_mins = int(elapsed_time / 60)
    elapsed_secs = int(elapsed_time - (elapsed_mins * 60))
    return elapsed_mins, elapsed_secs

N_EPOCHS = 10

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'wordavg-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

Epoch: 01 | Epoch Time: 0m 2s
    Train Loss: 0.685 | Train Acc: 56.84%
     Val. Loss: 0.622 |  Val. Acc: 71.09%
Epoch: 02 | Epoch Time: 0m 2s
    Train Loss: 0.642 | Train Acc: 71.31%
     Val. Loss: 0.510 |  Val. Acc: 75.48%
Epoch: 03 | Epoch Time: 0m 2s
    Train Loss: 0.573 | Train Acc: 78.31%
     Val. Loss: 0.449 |  Val. Acc: 79.52%
Epoch: 04 | Epoch Time: 0m 2s
    Train Loss: 0.503 | Train Acc: 82.78%
     Val. Loss: 0.419 |  Val. Acc: 82.72%
Epoch: 05 | Epoch Time: 0m 2s
    Train Loss: 0.440 | Train Acc: 85.84%
     Val. Loss: 0.408 |  Val. Acc: 84.75%
Epoch: 06 | Epoch Time: 0m 2s
    Train Loss: 0.389 | Train Acc: 87.59%
     Val. Loss: 0.413 |  Val. Acc: 86.02%
Epoch: 07 | Epoch Time: 0m 2s
    Train Loss: 0.352 | Train Acc: 88.85%
     Val. Loss: 0.425 |  Val. Acc: 86.92%
Epoch: 08 | Epoch Time: 0m 2s
    Train Loss: 0.320 | Train Acc: 89.93%
     Val. Loss: 0.440 |  Val. Acc: 87.54%
Epoch: 09 | Epoch Time: 0m 2s
    Train Loss: 0.294 | Train Acc: 90.74%
     Val. Loss: 0.456 |  Val. Acc: 88.09%
Epoch: 10 | Epoch Time: 0m 2s
    Train Loss: 0.274 | Train Acc: 91.27%
     Val. Loss: 0.468 |  Val. Acc: 88.49%

import spacy
nlp = spacy.load('en')

def predict_sentiment(sentence):
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    tensor = torch.LongTensor(indexed).to(device)
    tensor = tensor.unsqueeze(1)
    prediction = torch.sigmoid(model(tensor))
    return prediction.item()

1	predict_sentiment("This film is terrible")

5.568591932965664e-26

1	predict_sentiment("This film is great")

1.0

RNN模型

下面我们尝试把模型换成一个recurrent neural network (RNN)。RNN经常会被用来encode一个sequence
$$h_t = \text{RNN}(x_t, h_{t-1})$$
我们使用最后一个hidden state $h_T$来表示整个句子。
然后我们把$h_T$通过一个线性变换$f$，然后用来预测句子的情感。

class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, 
                 n_layers, bidirectional, dropout, pad_idx):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.rnn = nn.LSTM(embedding_dim, hidden_dim, num_layers=n_layers, 
                           bidirectional=bidirectional, dropout=dropout)
        self.fc = nn.Linear(hidden_dim*2, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        embedded = self.dropout(self.embedding(text)) #[sent len, batch size, emb dim]
        output, (hidden, cell) = self.rnn(embedded)
        #output = [sent len, batch size, hid dim * num directions]
        #hidden = [num layers * num directions, batch size, hid dim]
        #cell = [num layers * num directions, batch size, hid dim]
        
        #concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
        #and apply dropout
        hidden = self.dropout(torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim=1)) # [batch size, hid dim * num directions]
        return self.fc(hidden.squeeze(0))

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM, 
            N_LAYERS, BIDIRECTIONAL, DROPOUT, PAD_IDX)

1	print(f'The model has {count_parameters(model):,} trainable parameters')

The model has 4,810,857 trainable parameters

model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

print(model.embedding.weight.data)

tensor([[ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [ 0.0000,  0.0000,  0.0000,  ...,  0.0000,  0.0000,  0.0000],
        [-0.0382, -0.2449,  0.7281,  ..., -0.1459,  0.8278,  0.2706],
        ...,
        [-0.7244, -0.0186,  0.0996,  ...,  0.0045, -1.0037,  0.6646],
        [-1.1243,  1.2040, -0.6489,  ..., -0.7526,  0.5711,  1.0081],
        [ 0.0860,  0.1367,  0.0321,  ..., -0.5542, -0.4557, -0.0382]],
       device='cuda:0')

训练RNN模型

1 2	optimizer = optim.Adam(model.parameters()) model = model.to(device)

N_EPOCHS = 5
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'lstm-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

Epoch: 01 | Epoch Time: 1m 29s
    Train Loss: 0.676 | Train Acc: 57.69%
     Val. Loss: 0.694 |  Val. Acc: 53.40%
Epoch: 02 | Epoch Time: 1m 29s
    Train Loss: 0.641 | Train Acc: 63.77%
     Val. Loss: 0.744 |  Val. Acc: 49.22%
Epoch: 03 | Epoch Time: 1m 29s
    Train Loss: 0.618 | Train Acc: 65.77%
     Val. Loss: 0.534 |  Val. Acc: 73.72%
Epoch: 04 | Epoch Time: 1m 30s
    Train Loss: 0.634 | Train Acc: 63.79%
     Val. Loss: 0.619 |  Val. Acc: 66.85%
Epoch: 05 | Epoch Time: 1m 29s
    Train Loss: 0.448 | Train Acc: 79.19%
     Val. Loss: 0.340 |  Val. Acc: 86.63%

You may have noticed the loss is not really decreasing and the accuracy is poor. This is due to several issues with the model which we’ll improve in the next notebook.

Finally, the metric we actually care about, the test loss and accuracy, which we get from our parameters that gave us the best validation loss.

1
2
3

model.load_state_dict(torch.load('lstm-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

CNN模型

class CNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, n_filters, 
                 filter_sizes, output_dim, dropout, pad_idx):
        super().__init__()
        
        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)
        self.convs = nn.ModuleList([
                                    nn.Conv2d(in_channels = 1, out_channels = n_filters, 
                                              kernel_size = (fs, embedding_dim)) 
                                    for fs in filter_sizes
                                    ])
        self.fc = nn.Linear(len(filter_sizes) * n_filters, output_dim)
        self.dropout = nn.Dropout(dropout)
        
    def forward(self, text):
        text = text.permute(1, 0) # [batch size, sent len]
        embedded = self.embedding(text) # [batch size, sent len, emb dim]
        embedded = embedded.unsqueeze(1) # [batch size, 1, sent len, emb dim]
        conved = [F.relu(conv(embedded)).squeeze(3) for conv in self.convs]
            
        #conv_n = [batch size, n_filters, sent len - filter_sizes[n]]
        
        pooled = [F.max_pool1d(conv, conv.shape[2]).squeeze(2) for conv in conved]
        
        #pooled_n = [batch size, n_filters]
        
        cat = self.dropout(torch.cat(pooled, dim=1))

        #cat = [batch size, n_filters * len(filter_sizes)]
            
        return self.fc(cat)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
N_FILTERS = 100
FILTER_SIZES = [3,4,5]
OUTPUT_DIM = 1
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]


model = CNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM, DROPOUT, PAD_IDX)
model.embedding.weight.data.copy_(pretrained_embeddings)
UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)
model = model.to(device)

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()
criterion = criterion.to(device)

N_EPOCHS = 5

best_valid_loss = float('inf')

for epoch in range(N_EPOCHS):

    start_time = time.time()
    
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    
    end_time = time.time()

    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'CNN-model.pt')
    
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

Epoch: 01 | Epoch Time: 0m 11s
    Train Loss: 0.645 | Train Acc: 62.12%
     Val. Loss: 0.485 |  Val. Acc: 79.61%
Epoch: 02 | Epoch Time: 0m 11s
    Train Loss: 0.423 | Train Acc: 80.59%
     Val. Loss: 0.360 |  Val. Acc: 84.63%
Epoch: 03 | Epoch Time: 0m 11s
    Train Loss: 0.302 | Train Acc: 87.33%
     Val. Loss: 0.320 |  Val. Acc: 86.59%
Epoch: 04 | Epoch Time: 0m 11s
    Train Loss: 0.222 | Train Acc: 91.20%
     Val. Loss: 0.306 |  Val. Acc: 87.17%
Epoch: 05 | Epoch Time: 0m 11s
    Train Loss: 0.161 | Train Acc: 93.99%
     Val. Loss: 0.325 |  Val. Acc: 86.82%

1
2
3

model.load_state_dict(torch.load('CNN-model.pt'))
test_loss, test_acc = evaluate(model, test_iterator, criterion)
print(f'Test Loss: {test_loss:.3f} | Test Acc: {test_acc*100:.2f}%')

Test Loss: 0.336 | Test Acc: 85.66%

3.language-model

2020-02-29

第三课语言模型

褚则伟 zeweichu@gmail.com

学习目标

学习语言模型，以及如何训练一个语言模型
学习torchtext的基本使用方法
- 构建 vocabulary
- word to inde 和 index to word
学习torch.nn的一些基本模型
- Linear
- RNN
- LSTM
- GRU
RNN的训练技巧
- Gradient Clipping
如何保存和读取模型

我们会使用 torchtext 来创建vocabulary, 然后把数据读成batch的格式。请大家自行阅读README来学习torchtext。

import torchtext
from torchtext.vocab import Vectors
import torch
import numpy as np
import random

USE_CUDA = torch.cuda.is_available()

# 为了保证实验结果可以复现，我们经常会把各种random seed固定在某一个值
random.seed(53113)
np.random.seed(53113)
torch.manual_seed(53113)
if USE_CUDA:
    torch.cuda.manual_seed(53113)

BATCH_SIZE = 32
EMBEDDING_SIZE = 650
MAX_VOCAB_SIZE = 50000

我们会继续使用上次的text8作为我们的训练，验证和测试数据
TorchText的一个重要概念是Field，它决定了你的数据会如何被处理。我们使用TEXT这个field来处理文本数据。我们的TEXT field有lower=True这个参数，所以所有的单词都会被lowercase。
torchtext提供了LanguageModelingDataset这个class来帮助我们处理语言模型数据集。
build_vocab可以根据我们提供的训练数据集来创建最高频单词的单词表，max_size帮助我们限定单词总量。
BPTTIterator可以连续地得到连贯的句子，BPTT的全程是back propagation through time。

TEXT = torchtext.data.Field(lower=True)
train, val, test = torchtext.datasets.LanguageModelingDataset.splits(path=".", 
    train="text8.train.txt", validation="text8.dev.txt", test="text8.test.txt", text_field=TEXT)
TEXT.build_vocab(train, max_size=MAX_VOCAB_SIZE)
print("vocabulary size: {}".format(len(TEXT.vocab)))

VOCAB_SIZE = len(TEXT.vocab)
train_iter, val_iter, test_iter = torchtext.data.BPTTIterator.splits(
    (train, val, test), batch_size=BATCH_SIZE, device=-1, bptt_len=32, repeat=False, shuffle=True)

The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.


vocabulary size: 50002

为什么我们的单词表有50002个单词而不是50000呢？因为TorchText给我们增加了两个特殊的token，<unk>表示未知的单词，<pad>表示padding。
模型的输入是一串文字，模型的输出也是一串文字，他们之间相差一个位置，因为语言模型的目标是根据之前的单词预测下一个单词。

it = iter(train_iter)
batch = next(it)
print(" ".join([TEXT.vocab.itos[i] for i in batch.text[:,1].data]))
print(" ".join([TEXT.vocab.itos[i] for i in batch.target[:,1].data]))

had dropped to just three zero zero zero k it was then cool enough to allow the nuclei to capture electrons this process is called recombination during which the first neutral atoms
dropped to just three zero zero zero k it was then cool enough to allow the nuclei to capture electrons this process is called recombination during which the first neutral atoms took

for i in range(5):
    batch = next(it)
    print(" ".join([TEXT.vocab.itos[i] for i in batch.text[:,2].data]))
    print(" ".join([TEXT.vocab.itos[i] for i in batch.target[:,2].data]))

under the aegis of the emperor thus reducing the local identity and autonomy of the different regions of japan as japanese citizens the ainu are now governed by japanese laws though one
the aegis of the emperor thus reducing the local identity and autonomy of the different regions of japan as japanese citizens the ainu are now governed by japanese laws though one ainu
ainu man was acquitted of murder because he asserted that he was not a japanese citizen and the judge agreed and judged by japanese tribunals but in the past their affairs were
man was acquitted of murder because he asserted that he was not a japanese citizen and the judge agreed and judged by japanese tribunals but in the past their affairs were administered
administered by hereditary chiefs three in each village and for administrative purposes the country was divided into three districts <unk> <unk> and <unk> which were under the ultimate control of <unk> though
by hereditary chiefs three in each village and for administrative purposes the country was divided into three districts <unk> <unk> and <unk> which were under the ultimate control of <unk> though the
the relations between their respective inhabitants were not close and intermarriages were avoided the functions of judge were not entrusted to these chiefs an indefinite number of a community s members sat
relations between their respective inhabitants were not close and intermarriages were avoided the functions of judge were not entrusted to these chiefs an indefinite number of a community s members sat in
in judgement upon its criminals capital punishment did not exist nor was imprisonment resorted to beating being considered a sufficient and final penalty except in the case of murder when the nose
judgement upon its criminals capital punishment did not exist nor was imprisonment resorted to beating being considered a sufficient and final penalty except in the case of murder when the nose and

定义模型

继承nn.Module
初始化函数
forward函数
其余可以根据模型需要定义相关的函数

import torch
import torch.nn as nn


class RNNModel(nn.Module):
    """ 一个简单的循环神经网络"""

    def __init__(self, rnn_type, ntoken, ninp, nhid, nlayers, dropout=0.5):
        ''' 该模型包含以下几层:
            - 词嵌入层
            - 一个循环神经网络层(RNN, LSTM, GRU)
            - 一个线性层，从hidden state到输出单词表
            - 一个dropout层，用来做regularization
        '''
        super(RNNModel, self).__init__()
        self.drop = nn.Dropout(dropout)
        self.encoder = nn.Embedding(ntoken, ninp)
        if rnn_type in ['LSTM', 'GRU']:
            self.rnn = getattr(nn, rnn_type)(ninp, nhid, nlayers, dropout=dropout)
        else:
            try:
                nonlinearity = {'RNN_TANH': 'tanh', 'RNN_RELU': 'relu'}[rnn_type]
            except KeyError:
                raise ValueError( """An invalid option for `--model` was supplied,
                                 options are ['LSTM', 'GRU', 'RNN_TANH' or 'RNN_RELU']""")
            self.rnn = nn.RNN(ninp, nhid, nlayers, nonlinearity=nonlinearity, dropout=dropout)
        self.decoder = nn.Linear(nhid, ntoken)

        self.init_weights()

        self.rnn_type = rnn_type
        self.nhid = nhid
        self.nlayers = nlayers

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, input, hidden):
        ''' Forward pass:
            - word embedding
            - 输入循环神经网络
            - 一个线性层从hidden state转化为输出单词表
        '''
        emb = self.drop(self.encoder(input))
        output, hidden = self.rnn(emb, hidden)
        output = self.drop(output)
        decoded = self.decoder(output.view(output.size(0)*output.size(1), output.size(2)))
        return decoded.view(output.size(0), output.size(1), decoded.size(1)), hidden

    def init_hidden(self, bsz, requires_grad=True):
        weight = next(self.parameters())
        if self.rnn_type == 'LSTM':
            return (weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad),
                    weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad))
        else:
            return weight.new_zeros((self.nlayers, bsz, self.nhid), requires_grad=requires_grad)

初始化一个模型

1
2
3

model = RNNModel("LSTM", VOCAB_SIZE, EMBEDDING_SIZE, EMBEDDING_SIZE, 2, dropout=0.5)
if USE_CUDA:
    model = model.cuda()

我们首先定义评估模型的代码。
模型的评估和模型的训练逻辑基本相同，唯一的区别是我们只需要forward pass，不需要backward pass

def evaluate(model, data):
    model.eval()
    total_loss = 0.
    it = iter(data)
    total_count = 0.
    with torch.no_grad():
        hidden = model.init_hidden(BATCH_SIZE, requires_grad=False)
        for i, batch in enumerate(it):
            data, target = batch.text, batch.target
            if USE_CUDA:
                data, target = data.cuda(), target.cuda()
            hidden = repackage_hidden(hidden)
            with torch.no_grad():
                output, hidden = model(data, hidden)
            loss = loss_fn(output.view(-1, VOCAB_SIZE), target.view(-1))
            total_count += np.multiply(*data.size())
            total_loss += loss.item()*np.multiply(*data.size())
            
    loss = total_loss / total_count
    model.train()
    return loss

我们需要定义下面的一个function，帮助我们把一个hidden state和计算图之前的历史分离。

# Remove this part
def repackage_hidden(h):
    """Wraps hidden states in new Tensors, to detach them from their history."""
    if isinstance(h, torch.Tensor):
        return h.detach()
    else:
        return tuple(repackage_hidden(v) for v in h)

定义loss function和optimizer

loss_fn = nn.CrossEntropyLoss()
learning_rate = 0.001
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer, 0.5)

训练模型：

模型一般需要训练若干个epoch
每个epoch我们都把所有的数据分成若干个batch
把每个batch的输入和输出都包装成cuda tensor
forward pass，通过输入的句子预测每个单词的下一个单词
用模型的预测和正确的下一个单词计算cross entropy loss
清空模型当前gradient
backward pass
gradient clipping，防止梯度爆炸
更新模型参数
每隔一定的iteration输出模型在当前iteration的loss，以及在验证集上做模型的评估

import copy
GRAD_CLIP = 1.
NUM_EPOCHS = 2

val_losses = []
for epoch in range(NUM_EPOCHS):
    model.train()
    it = iter(train_iter)
    hidden = model.init_hidden(BATCH_SIZE)
    for i, batch in enumerate(it):
        data, target = batch.text, batch.target
        if USE_CUDA:
            data, target = data.cuda(), target.cuda()
        hidden = repackage_hidden(hidden)
        model.zero_grad()
        output, hidden = model(data, hidden)
        loss = loss_fn(output.view(-1, VOCAB_SIZE), target.view(-1))
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), GRAD_CLIP)
        optimizer.step()
        if i % 1000 == 0:
            print("epoch", epoch, "iter", i, "loss", loss.item())
    
        if i % 10000 == 0:
            val_loss = evaluate(model, val_iter)
            
            if len(val_losses) == 0 or val_loss < min(val_losses):
                print("best model, val loss: ", val_loss)
                torch.save(model.state_dict(), "lm-best.th")
            else:
                scheduler.step()
                optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
            val_losses.append(val_loss)

epoch 0 iter 0 loss 10.821578979492188
best model, val loss:  10.782116411285918
epoch 0 iter 1000 loss 6.5122528076171875
epoch 0 iter 2000 loss 6.3599748611450195
epoch 0 iter 3000 loss 6.13856315612793
epoch 0 iter 4000 loss 5.473214626312256
epoch 0 iter 5000 loss 5.901871204376221
epoch 0 iter 6000 loss 5.85321569442749
epoch 0 iter 7000 loss 5.636535167694092
epoch 0 iter 8000 loss 5.7489800453186035
epoch 0 iter 9000 loss 5.464158058166504
epoch 0 iter 10000 loss 5.554863452911377
best model, val loss:  5.264891533569864
epoch 0 iter 11000 loss 5.703625202178955
epoch 0 iter 12000 loss 5.6448974609375
epoch 0 iter 13000 loss 5.372857570648193
epoch 0 iter 14000 loss 5.2639479637146
epoch 1 iter 0 loss 5.696778297424316
best model, val loss:  5.124550380139679
epoch 1 iter 1000 loss 5.534722805023193
epoch 1 iter 2000 loss 5.599489212036133
epoch 1 iter 3000 loss 5.459986686706543
epoch 1 iter 4000 loss 4.927192211151123
epoch 1 iter 5000 loss 5.435710906982422
epoch 1 iter 6000 loss 5.4059576988220215
epoch 1 iter 7000 loss 5.308575630187988
epoch 1 iter 8000 loss 5.405811786651611
epoch 1 iter 9000 loss 5.1389055252075195
epoch 1 iter 10000 loss 5.226413726806641
best model, val loss:  4.946829228873176
epoch 1 iter 11000 loss 5.379891395568848
epoch 1 iter 12000 loss 5.360724925994873
epoch 1 iter 13000 loss 5.176026344299316
epoch 1 iter 14000 loss 5.110936641693115

best_model = RNNModel("LSTM", VOCAB_SIZE, EMBEDDING_SIZE, EMBEDDING_SIZE, 2, dropout=0.5)
if USE_CUDA:
    best_model = best_model.cuda()
best_model.load_state_dict(torch.load("lm-best.th"))

使用最好的模型在valid数据上计算perplexity

1 2	val_loss = evaluate(best_model, val_iter) print("perplexity: ", np.exp(val_loss))

perplexity:  140.72803934425724

使用最好的模型在测试数据上计算perplexity

1 2	test_loss = evaluate(best_model, test_iter) print("perplexity: ", np.exp(test_loss))

perplexity:  178.54742013696125

使用训练好的模型生成一些句子。

hidden = best_model.init_hidden(1)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
input = torch.randint(VOCAB_SIZE, (1, 1), dtype=torch.long).to(device)
words = []
for i in range(100):
    output, hidden = best_model(input, hidden)
    word_weights = output.squeeze().exp().cpu()
    word_idx = torch.multinomial(word_weights, 1)[0]
    input.fill_(word_idx)
    word = TEXT.vocab.itos[word_idx]
    words.append(word)
print(" ".join(words))

s influence clinton decision de gaulle is himself sappho s iv one family banquet was made published by paul <unk> and by a persuaded to prevent arcane of animate poverty based at copernicus bachelor in search services and in a cruise corps references eds the robin series july four one nine zero eight summer gutenberg one nine six four births one nine two eight deaths timeline of this method by the fourth amendment the german ioc known for his <unk> from <unk> one eight nine eight one seven eight nine management was established in one nine seven zero they had

2.word-embedding

2020-02-29

第二课词向量

褚则伟 zeweichu@gmail.com

第二课学习目标

学习词向量的概念
用Skip-thought模型训练词向量
学习使用PyTorch dataset和dataloader
学习定义PyTorch模型
学习torch.nn中常见的Module
- Embedding
学习常见的PyTorch operations
- bmm
- logsigmoid
保存和读取PyTorch模型

第二课使用的训练数据可以从以下链接下载到。

链接:https://pan.baidu.com/s/1tFeK3mXuVXEy3EMarfeWvg 密码:v2z5

在这一份notebook中，我们会（尽可能）尝试复现论文Distributed Representations of Words and Phrases and their Compositionality中训练词向量的方法. 我们会实现Skip-gram模型，并且使用论文中noice contrastive sampling的目标函数。

这篇论文有很多模型实现的细节，这些细节对于词向量的好坏至关重要。我们虽然无法完全复现论文中的实验结果，主要是由于计算资源等各种细节原因，但是我们还是可以大致展示如何训练词向量。

以下是一些我们没有实现的细节

subsampling：参考论文section 2.3

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data as tud
from torch.nn.parameter import Parameter

from collections import Counter
import numpy as np
import random
import math

import pandas as pd
import scipy
import sklearn
from sklearn.metrics.pairwise import cosine_similarity

USE_CUDA = torch.cuda.is_available()

# 为了保证实验结果可以复现，我们经常会把各种random seed固定在某一个值
random.seed(53113)
np.random.seed(53113)
torch.manual_seed(53113)
if USE_CUDA:
    torch.cuda.manual_seed(53113)
    
# 设定一些超参数
    
K = 100 # number of negative samples
C = 3 # nearby words threshold
NUM_EPOCHS = 2 # The number of epochs of training
MAX_VOCAB_SIZE = 30000 # the vocabulary size
BATCH_SIZE = 128 # the batch size
LEARNING_RATE = 0.2 # the initial learning rate
EMBEDDING_SIZE = 100
       
    
LOG_FILE = "word-embedding.log"

# tokenize函数，把一篇文本转化成一个个单词
def word_tokenize(text):
    return text.split()

从文本文件中读取所有的文字，通过这些文本创建一个vocabulary
由于单词数量可能太大，我们只选取最常见的MAX_VOCAB_SIZE个单词
我们添加一个UNK单词表示所有不常见的单词
我们需要记录单词到index的mapping，以及index到单词的mapping，单词的count，单词的(normalized) frequency，以及单词总数。

with open("./lesson2/text8.train.txt", "r") as fin:
    text = fin.read()
    
text = [w for w in word_tokenize(text.lower())]
vocab = dict(Counter(text).most_common(MAX_VOCAB_SIZE-1)) # map to wordcount (MAX_VOCAB_SIZE-1 : except <unk>)
vocab["<unk>"] = len(text) - np.sum(list(vocab.values()))
# two-way mapping
idx_to_word = [word for word in vocab.keys()] 
word_to_idx = {word:i for i, word in enumerate(idx_to_word)}

word_counts = np.array([count for count in vocab.values()], dtype=np.float32)
word_freqs = word_counts / np.sum(word_counts)
word_freqs = word_freqs ** (3./4.)
word_freqs = word_freqs / np.sum(word_freqs) # 用来做 negative sampling
VOCAB_SIZE = len(idx_to_word)
VOCAB_SIZE

实现Dataloader

一个dataloader需要以下内容：

把所有text编码成数字，然后用subsampling预处理这些文字。
保存vocabulary，单词count，normalized word frequency
每个iteration sample一个中心词
根据当前的中心词返回context单词
根据中心词sample一些negative单词
返回单词的counts

这里有一个好的tutorial介绍如何使用PyTorch dataloader.
为了使用dataloader，我们需要定义以下两个function:

__len__ function需要返回整个数据集中有多少个item
__get__ 根据给定的index返回一个item

有了dataloader之后，我们可以轻松随机打乱整个数据集，拿到一个batch的数据等等。

class WordEmbeddingDataset(tud.Dataset):
    def __init__(self, text, word_to_idx, idx_to_word, word_freqs, word_counts):
        ''' text: a list of words, all text from the training dataset
            word_to_idx: the dictionary from word to idx
            idx_to_word: idx to word mapping
            word_freq: the frequency of each word
            word_counts: the word counts
        '''
        super(WordEmbeddingDataset, self).__init__()
        self.text_encoded = [word_to_idx.get(t, VOCAB_SIZE-1) for t in text]
        self.text_encoded = torch.Tensor(self.text_encoded).long()
        self.word_to_idx = word_to_idx
        self.idx_to_word = idx_to_word
        self.word_freqs = torch.Tensor(word_freqs)
        self.word_counts = torch.Tensor(word_counts)
        
    def __len__(self):
        ''' 返回整个数据集（所有单词）的长度
        '''
        return len(self.text_encoded)
        
    def __getitem__(self, idx):
        ''' 这个function返回以下数据用于训练
            - 中心词
            - 这个单词附近的(positive)单词
            - 随机采样的K个单词作为negative sample
        '''
        center_word = self.text_encoded[idx]
        pos_indices = list(range(idx-C, idx)) + list(range(idx+1, idx+C+1))
        pos_indices = [i%len(self.text_encoded) for i in pos_indices]
        pos_words = self.text_encoded[pos_indices] 
        neg_words = torch.multinomial(self.word_freqs, K * pos_words.shape[0], True)
        
        return center_word, pos_words, neg_words

创建dataset和dataloader

1 2	dataset = WordEmbeddingDataset(text, word_to_idx, idx_to_word, word_freqs, word_counts) dataloader = tud.DataLoader(dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=0)

1	next(iter(dataloader))

[tensor([   48,  1495,  1799,  2312,  2329,  4869,     8,     0,  8746,    71,
           772,     7,  2837,     0, 11805,  1746,  7122,     6,    10,  1956,
          9429,    33,    17,   230, 29999,     5,   291,   640, 13454,   103,
          2505,   540,  1683,    25,   330,    52,  1890,   644,   219,     6,
         29999,    82, 27040,  2008,     1,  1832,  5685,     0,    53,  1214,
            58, 29999,  7677,     9,    14,     5,  7035, 12651,    73,   788,
          1159,     7,     0,    40,     3, 29999,    53,    57,   714,  5871,
          1056,  2756,     5,  5272,    26, 29999,    44,     8,    35,  6009,
            41,     5,   692,    17,   854,  2473,     4,    33,     1, 12482,
            48,     7,    30,   306,  7595,     6,     5,    56,  1461,     2,
           131,   167,   232,    11,  9131, 18990,     4,  4395,    32,    19,
             0,   847,    17,  6116,   608,  2573,  6378, 29999,    52, 17768,
            20,     0,  4113,  8883,  2380,   121,  1298,    87]),
 tensor([[  480,     0,   868,   363,    13,    93],
         [ 5889, 10403,  3751,  5168,   932, 16947],
         [  749,     3,     5,   345,     1,     0],
         [ 3798,  8075,     2,  1008,     2,    50],
         [ 5607, 19200,  3981,  1436,   582,     4],
         [ 2300,     4,     5,   213,    27,  1122],
         [   14,  1761,     3,    22,     8,  2573],
         [ 8123,  2813,    19,  9093,  2077,    96],
         [   13,    44,  2189,  5529,    26,    17],
         [ 2949,     0,  2266,  4809,  2876,   169],
         [ 2084,  4219,  1173,  1262, 29999,   772],
         [ 3284,     9,     7,     7,   599,   486],
         [29999,     5,  2252, 14789,     0,  7853],
         [   83,    70,    38,  4720,     1,  6251],
         [    0,  4719,     2,    39,  2117,     6],
         [   13,   254,   387,    23,     0,   121],
         [  266,    54,    11,  6962,  4473,     2],
         [  569,    10,  7479,  5115, 17946,    27],
         [    8,    32,   557,   657,     5,   297],
         [10982,   216,     6,  1051,  4695,   130],
         [ 6260,  1463,     6,     0, 11117,  1174],
         [ 8131, 16819,     1,     0, 29999,   695],
         [   11,     5,  9742,     4,     0,   490],
         [   14,   227,  6220,     5,     2,  2834],
         [ 8529,     1,     0,   212,     0,   314],
         [   20,    33,  6362,  5836,     4,    94],
         [   32,    10,  1305,   174,     0, 29999],
         [   38,   724,    19,     4,     0,   770],
         [ 9207,  5098,     2,     1,  1360,   105],
         [   14,     0, 29999,  7322,     6, 19459],
         [  974,     4,   864,     2,     0,   113],
         [   55,  5216,  2214,  1314,    80,     2],
         [    2, 26169,  1027,  2814,    24,   977],
         [  667,     1, 17338,  1043,    27,     0],
         [   18,  1574,  3048,   184,    66,   617],
         [   27,   280,   577,  1202,     0, 19816],
         [    5,  5331,     6,    26,   275,    23],
         [ 2638,     0,   219,    67,    37,   988],
         [   44,   890,    18,    32,   776, 27004],
         [    4,     0, 15733,  1790,     5,   953],
         [    6,  6495,     0,  1162,     2,  1316],
         [    3,    15,   346,     4,     0,  3131],
         [   11,    45,   136,    27,   135,    42],
         [ 5191,    24,     5, 14534,   481, 29282],
         [   10,     0,   181,   973,    19,    25],
         [  144,  1288,  3860,    30,  3778,     1],
         [  973, 10080,  1355,    25, 13110,    27],
         [    5,    62,    23,   358,  5077, 10844],
         [ 1778,    35,    48,   223,  1459,    13],
         [    0,   477,   158,  5005,    19,     0],
         [ 4364,  2567,    91,    25,   280,  2567],
         [29999, 24131,   300, 29999, 21067,   173],
         [   12, 29999, 29999, 29999, 29999,    14],
         [  145,     2,    29,  2020,   145,  2680],
         [ 4764,  2094, 12947,  3593,     2,     0],
         [   13,   168, 10474, 29999,    10,     5],
         [ 1011,  4210, 29999, 29999,     3,     3],
         [    9,     9,  2471,     3,    22,   227],
         [  227,     5,  5760,   227,     5,   227],
         [   26,    51,   154,     5,     3,   346],
         [  653,     0,   439,     2,   324,    44],
         [ 2179,    13,    20,     1,  1598,     2],
         [29999,   960,     1,  1098,   260, 11907],
         [  396, 29999,    26,    53,  1667,    19],
         [ 3245,  1027,     4,     8,    21,    22],
         [    2,  4557,     1,   222,   445,  8796],
         [ 3672,    19,    48,   292,   309,    70],
         [  918,    27,     0,  1384,     6,  4626],
         [   76,    59,     0,  1986,    34,    43],
         [    8,    12,  2702, 25944,   126,  2825],
         [   19,   132,   686,  1403,     1,     0],
         [ 1390,   348,  6753,     0, 14359, 12643],
         [    1,   356,    10,   951,    13,  5075],
         [12655,  8176, 29999,     5, 18003,   125],
         [ 3232,     1,   258,    10,   106,  1348],
         [   12, 29999,  1769,    74, 29999,   164],
         [  185,  1282,   624,  2038,  9707,    26],
         [ 3149,    73,     3,     7,     9,     3],
         [    6,  5528,   187,    48,   197,     0],
         [  597,     1,  5820,    14,     0,   898],
         [   32,   479,   259,  2685,   310,   385],
         [ 2542,    58,    10,   717,     4,  3663],
         [    1,    65,     9,  2206,     6,    38],
         [ 4089,  3286,  1475,     3,     1,     0],
         [    0,   838,   551,   136,   175,     0],
         [  267,    78,     0,    67, 29999,    20],
         [    7,     7, 29944,     0,     3,     8],
         [ 1883,   141,    24,     6,  6214,  1367],
         [  107,  5667,    89,     0,  1408,    35],
         [  156,    75,    32,     0,   774,     6],
         [   19,    35,   385,   643,     2,    19],
         [ 4823, 10775,    15,  1937,     2,  4839],
         [   24, 29999,    23,   362,  3762,   176],
         [ 2409,   268,     5,  3131,   209,    79],
         [ 9573,     1,   436, 18136,  7979,  5383],
         [  272,   242,   579,    11,   272,     3],
         [   11,   120,     1,   461,    25,    36],
         [   17,    74,     3, 28426,    13,   102],
         [    4,     0, 10960,     1, 25912,  1561],
         [  924,  3397,  1889,  1141,   843, 23654],
         [ 1950,     6,     0,  3384,  7379,    19],
         [   13,     0,  5386,    36,  1642,   275],
         [11913,   612,   768,     6,    44,  8895],
         [   51,    31,  3135,  1982,    19,     0],
         [    0,  2635,    27,   308,    48,  1335],
         [  343,  8573,    28,    32,  4394, 25742],
         [  125,   489,   767,    14,    98,     5],
         [  451,  1699, 21841, 29999,    56,   327],
         [    4,  3932,  1230, 11823,     3,     1],
         [  714,  5428,  2211,     0,   411,     1],
         [19968,     2,    18,  8396,  2094,  6346],
         [  222,  1333,  2301,   300,  2381,     2],
         [  305,    93,  1993,  5967,    63, 29999],
         [  916,    50,  1921,    25,  1277,  1539],
         [  312,    14,   413,    10,   147,  6320],
         [    7,     7,    15,   879,     9,     7],
         [    0,   200, 29999,     1,    50,     0],
         [    1, 22168,     4,  1523,  6396,   995],
         [22526,    47,    81,     1,  1632,    10],
         [ 2135,    10,  2059,   516,    89,     5],
         [ 2079,   909,    10,     7,     7,  2588],
         [ 6259,  7404,    10,   297,  2703,   628],
         [    5, 29999,   244,  2653,    13,    29],
         [ 1532,    23,     0,  7033,    76,  1174],
         [   26,   263,    36, 29999,    27,     0],
         [  674,   105,     0,     1,     0, 27222],
         [   51,    76,    31,   335,     2, 29999],
         [12187,     1,     0,   347,  8817, 29999]]),
 tensor([[ 6313, 23579,  1926,  ...,    64,   147,  1924],
         [12277,   523,   231,  ..., 19350, 17264,  8771],
         [  577, 27195, 11445,  ..., 27978, 19518,    59],
         ...,
         [  246,  7365,   280,  ...,  3584,    56,  3854],
         [    8, 24712,   125,  ..., 26156, 18621,   295],
         [25034,  5372,  5070,  ...,   965,  1358, 25900]])]

定义PyTorch模型

class EmbeddingModel(nn.Module):
    def __init__(self, vocab_size, embed_size):
        ''' 初始化输出和输出embedding
        '''
        super(EmbeddingModel, self).__init__()
        self.vocab_size = vocab_size
        self.embed_size = embed_size
        
        initrange = 0.5 / self.embed_size
        self.out_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)
        self.out_embed.weight.data.uniform_(-initrange, initrange)
        
        
        self.in_embed = nn.Embedding(self.vocab_size, self.embed_size, sparse=False)
        self.in_embed.weight.data.uniform_(-initrange, initrange)
        
        
    def forward(self, input_labels, pos_labels, neg_labels):
        '''
        input_labels: 中心词, [batch_size]
        pos_labels: 中心词周围 context window 出现过的单词 [batch_size * (window_size * 2)]
        neg_labelss: 中心词周围没有出现过的单词，从 negative sampling 得到 [batch_size, (window_size * 2 * K)]
        
        return: loss, [batch_size]
        '''
        
        batch_size = input_labels.size(0)
        
        input_embedding = self.in_embed(input_labels) # B * embed_size
        pos_embedding = self.out_embed(pos_labels) # B * (2*C) * embed_size
        neg_embedding = self.out_embed(neg_labels) # B * (2*C * K) * embed_size
      
        log_pos = torch.bmm(pos_embedding, input_embedding.unsqueeze(2)).squeeze() # B * (2*C)
        log_neg = torch.bmm(neg_embedding, -input_embedding.unsqueeze(2)).squeeze() # B * (2*C*K)

        log_pos = F.logsigmoid(log_pos).sum(1)
        log_neg = F.logsigmoid(log_neg).sum(1) # batch_size
       
        loss = log_pos + log_neg
        
        return -loss
    
    def input_embeddings(self):
        return self.in_embed.weight.data.cpu().numpy()

定义一个模型以及把模型移动到GPU

1
2
3

model = EmbeddingModel(VOCAB_SIZE, EMBEDDING_SIZE)
if USE_CUDA:
    model = model.cuda()

下面是评估模型的代码，以及训练模型的代码

def evaluate(filename, embedding_weights): 
    if filename.endswith(".csv"):
        data = pd.read_csv(filename, sep=",")
    else:
        data = pd.read_csv(filename, sep="\t")
    human_similarity = []
    model_similarity = []
    for i in data.iloc[:, 0:2].index:
        word1, word2 = data.iloc[i, 0], data.iloc[i, 1]
        if word1 not in word_to_idx or word2 not in word_to_idx:
            continue
        else:
            word1_idx, word2_idx = word_to_idx[word1], word_to_idx[word2]
            word1_embed, word2_embed = embedding_weights[[word1_idx]], embedding_weights[[word2_idx]]
            model_similarity.append(float(sklearn.metrics.pairwise.cosine_similarity(word1_embed, word2_embed)))
            human_similarity.append(float(data.iloc[i, 2]))

    return scipy.stats.spearmanr(human_similarity, model_similarity)# , model_similarity

def find_nearest(word):
    index = word_to_idx[word]
    embedding = embedding_weights[index]
    cos_dis = np.array([scipy.spatial.distance.cosine(e, embedding) for e in embedding_weights])
    return [idx_to_word[i] for i in cos_dis.argsort()[:10]]

训练模型：

模型一般需要训练若干个epoch
每个epoch我们都把所有的数据分成若干个batch
把每个batch的输入和输出都包装成cuda tensor
forward pass，通过输入的句子预测每个单词的下一个单词
用模型的预测和正确的下一个单词计算cross entropy loss
清空模型当前gradient
backward pass
更新模型参数
每隔一定的iteration输出模型在当前iteration的loss，以及在验证数据集上做模型的评估

optimizer = torch.optim.SGD(model.parameters(), lr=LEARNING_RATE)
for e in range(NUM_EPOCHS):
    for i, (input_labels, pos_labels, neg_labels) in enumerate(dataloader):
        
        
        # TODO
        input_labels = input_labels.long()
        pos_labels = pos_labels.long()
        neg_labels = neg_labels.long()
        if USE_CUDA:
            input_labels = input_labels.cuda()
            pos_labels = pos_labels.cuda()
            neg_labels = neg_labels.cuda()
            
        optimizer.zero_grad()
        loss = model(input_labels, pos_labels, neg_labels).mean()
        loss.backward()
        optimizer.step()

        if i % 100 == 0:
            with open(LOG_FILE, "a") as fout:
                fout.write("epoch: {}, iter: {}, loss: {}\n".format(e, i, loss.item()))
                print("epoch: {}, iter: {}, loss: {}".format(e, i, loss.item()))
            
        
        if i % 2000 == 0:
            embedding_weights = model.input_embeddings()
            sim_simlex = evaluate("simlex-999.txt", embedding_weights)
            sim_men = evaluate("men.txt", embedding_weights)
            sim_353 = evaluate("wordsim353.csv", embedding_weights)
            with open(LOG_FILE, "a") as fout:
                print("epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\n".format(
                    e, i, sim_simlex, sim_men, sim_353, find_nearest("monster")))
                fout.write("epoch: {}, iteration: {}, simlex-999: {}, men: {}, sim353: {}, nearest to monster: {}\n".format(
                    e, i, sim_simlex, sim_men, sim_353, find_nearest("monster")))
                
    embedding_weights = model.input_embeddings()
    np.save("embedding-{}".format(EMBEDDING_SIZE), embedding_weights)
    torch.save(model.state_dict(), "embedding-{}.th".format(EMBEDDING_SIZE))

epoch: 0, iter: 0, loss: 420.04736328125
epoch: 0, iteration: 0, simlex-999: SpearmanrResult(correlation=0.002806243285464091, pvalue=0.9309107582703205), men: SpearmanrResult(correlation=-0.03578915454199749, pvalue=0.06854012381329619), sim353: SpearmanrResult(correlation=0.02468906830123471, pvalue=0.6609497549092586), nearest to monster: ['monster', 'communism', 'bosses', 'microprocessors', 'infectious', 'debussy', 'unesco', 'tantamount', 'offices', 'tischendorf']

epoch: 0, iter: 100, loss: 278.9967041015625
epoch: 0, iter: 200, loss: 248.71990966796875
epoch: 0, iter: 300, loss: 202.95816040039062
epoch: 0, iter: 400, loss: 157.04776000976562
epoch: 0, iter: 500, loss: 137.83531188964844
epoch: 0, iter: 600, loss: 121.03585815429688
epoch: 0, iter: 700, loss: 105.300537109375
epoch: 0, iter: 800, loss: 114.10055541992188
epoch: 0, iter: 900, loss: 104.72723388671875
epoch: 0, iter: 1000, loss: 99.03569030761719
epoch: 0, iter: 1100, loss: 95.2179946899414
epoch: 0, iter: 1200, loss: 84.12557983398438
epoch: 0, iter: 1300, loss: 88.07209777832031
epoch: 0, iter: 1400, loss: 70.44454193115234
epoch: 0, iter: 1500, loss: 79.83641052246094
epoch: 0, iter: 1600, loss: 81.7451171875
epoch: 0, iter: 1700, loss: 75.91305541992188
epoch: 0, iter: 1800, loss: 65.86140441894531
epoch: 0, iter: 1900, loss: 69.81714630126953
epoch: 0, iter: 2000, loss: 71.05166625976562
epoch: 0, iteration: 2000, simlex-999: SpearmanrResult(correlation=-0.011490367338787073, pvalue=0.7225847577400916), men: SpearmanrResult(correlation=0.05671509287050605, pvalue=0.0038790264864563434), sim353: SpearmanrResult(correlation=-0.07381419228558825, pvalue=0.18921537418718104), nearest to monster: ['monster', 'harm', 'steel', 'dean', 'kansas', 'surgery', 'regardless', 'capitalism', 'offers', 'hockey']

epoch: 0, iter: 2100, loss: 59.19840621948242
epoch: 0, iter: 2200, loss: 60.21418762207031
epoch: 0, iter: 2300, loss: 63.848148345947266
epoch: 0, iter: 2400, loss: 65.58479309082031
epoch: 0, iter: 2500, loss: 66.90382385253906
epoch: 0, iter: 2600, loss: 54.61847686767578
epoch: 0, iter: 2700, loss: 56.45966339111328
epoch: 0, iter: 2800, loss: 58.255210876464844
epoch: 0, iter: 2900, loss: 59.65287399291992
epoch: 0, iter: 3000, loss: 48.22801971435547
epoch: 0, iter: 3100, loss: 42.94969177246094
epoch: 0, iter: 3200, loss: 49.372528076171875
epoch: 0, iter: 3300, loss: 46.12495422363281
epoch: 0, iter: 3400, loss: 58.97121047973633
epoch: 0, iter: 3500, loss: 48.31055450439453
epoch: 0, iter: 3600, loss: 47.07227325439453
epoch: 0, iter: 3700, loss: 46.4068603515625
epoch: 0, iter: 3800, loss: 49.55707931518555
epoch: 0, iter: 3900, loss: 44.38733673095703
epoch: 0, iter: 4000, loss: 48.730342864990234
epoch: 0, iteration: 4000, simlex-999: SpearmanrResult(correlation=0.0190424235850696, pvalue=0.5562848091306694), men: SpearmanrResult(correlation=0.05404895260610133, pvalue=0.00592548586032086), sim353: SpearmanrResult(correlation=-0.039572591538143916, pvalue=0.4819454801463242), nearest to monster: ['monster', 'electrical', 'northeast', 'surgery', 'entity', 'certainly', 'tea', 'establishing', 'archbishop', 'aging']

epoch: 0, iter: 4100, loss: 57.70344161987305
epoch: 0, iter: 4200, loss: 47.464820861816406
epoch: 0, iter: 4300, loss: 47.08036804199219
epoch: 0, iter: 4400, loss: 46.652706146240234
epoch: 0, iter: 4500, loss: 40.824310302734375
epoch: 0, iter: 4600, loss: 40.62211227416992
epoch: 0, iter: 4700, loss: 50.84752655029297
epoch: 0, iter: 4800, loss: 41.230072021484375
epoch: 0, iter: 4900, loss: 53.74473571777344
epoch: 0, iter: 5000, loss: 42.35053253173828
epoch: 0, iter: 5100, loss: 38.363189697265625
epoch: 0, iter: 5200, loss: 42.772552490234375
epoch: 0, iter: 5300, loss: 44.914913177490234
epoch: 0, iter: 5400, loss: 38.4688720703125
epoch: 0, iter: 5500, loss: 41.0843391418457
epoch: 0, iter: 5600, loss: 35.04629898071289
epoch: 0, iter: 5700, loss: 35.49506759643555
epoch: 0, iter: 5800, loss: 36.009666442871094
epoch: 0, iter: 5900, loss: 40.56498718261719
epoch: 0, iter: 6000, loss: 45.853214263916016
epoch: 0, iteration: 6000, simlex-999: SpearmanrResult(correlation=0.04213372810279324, pvalue=0.19281410892481102), men: SpearmanrResult(correlation=0.06483263975087832, pvalue=0.0009600352172924885), sim353: SpearmanrResult(correlation=-0.015385630136134733, pvalue=0.7846219761829791), nearest to monster: ['monster', 'raw', 'romantic', 'oregon', 'protest', 'brunei', 'cartoon', 'offers', 'certainly', 'ill']

epoch: 0, iter: 6100, loss: 39.977508544921875
epoch: 0, iter: 6200, loss: 35.47979736328125
epoch: 0, iter: 6300, loss: 38.61311340332031
epoch: 0, iter: 6400, loss: 38.735679626464844
epoch: 0, iter: 6500, loss: 41.1725959777832
epoch: 0, iter: 6600, loss: 37.390037536621094
epoch: 0, iter: 6700, loss: 39.51911926269531
epoch: 0, iter: 6800, loss: 47.12213897705078
epoch: 0, iter: 6900, loss: 41.91630172729492
epoch: 0, iter: 7000, loss: 38.11504364013672
epoch: 0, iter: 7100, loss: 38.12763214111328
epoch: 0, iter: 7200, loss: 36.93813705444336
epoch: 0, iter: 7300, loss: 40.82877731323242
epoch: 0, iter: 7400, loss: 36.211429595947266
epoch: 0, iter: 7500, loss: 36.141693115234375
epoch: 0, iter: 7600, loss: 38.152610778808594
epoch: 0, iter: 7700, loss: 38.90789031982422
epoch: 0, iter: 7800, loss: 36.30712127685547
epoch: 0, iter: 7900, loss: 34.192440032958984
epoch: 0, iter: 8000, loss: 39.182212829589844
epoch: 0, iteration: 8000, simlex-999: SpearmanrResult(correlation=0.05506138271487322, pvalue=0.0886781241789579), men: SpearmanrResult(correlation=0.06796632118931804, pvalue=0.0005362832465382729), sim353: SpearmanrResult(correlation=-0.00727317983344893, pvalue=0.897207043425527), nearest to monster: ['monster', 'raw', 'romantic', 'strategic', 'offers', 'invited', 'signature', 'piano', 'protest', 'bills']

epoch: 0, iter: 8100, loss: 35.08313751220703
epoch: 0, iter: 8200, loss: 33.23561096191406
epoch: 0, iter: 8300, loss: 36.047096252441406
epoch: 0, iter: 8400, loss: 37.01750946044922
epoch: 0, iter: 8500, loss: 33.679561614990234
epoch: 0, iter: 8600, loss: 36.492515563964844
epoch: 0, iter: 8700, loss: 34.439537048339844
epoch: 0, iter: 8800, loss: 38.89817428588867
epoch: 0, iter: 8900, loss: 34.17725372314453
epoch: 0, iter: 9000, loss: 33.869651794433594
epoch: 0, iter: 9100, loss: 33.63176727294922
epoch: 0, iter: 9200, loss: 35.203460693359375
epoch: 0, iter: 9300, loss: 36.060142517089844
epoch: 0, iter: 9400, loss: 35.6544303894043
epoch: 0, iter: 9500, loss: 35.01182556152344
epoch: 0, iter: 9600, loss: 35.48432540893555
epoch: 0, iter: 9700, loss: 34.940696716308594
epoch: 0, iter: 9800, loss: 33.99235534667969
epoch: 0, iter: 9900, loss: 35.14078903198242
epoch: 0, iter: 10000, loss: 34.10219192504883
epoch: 0, iteration: 10000, simlex-999: SpearmanrResult(correlation=0.0714732189475033, pvalue=0.02703637716635098), men: SpearmanrResult(correlation=0.07013186360584196, pvalue=0.00035356323424747736), sim353: SpearmanrResult(correlation=-0.0013966072615024432, pvalue=0.9802088977698729), nearest to monster: ['monster', 'adoption', 'logo', 'particle', 'isle', 'remainder', 'profit', 'rank', 'execution', 'outer']

epoch: 0, iter: 10100, loss: 33.885284423828125
epoch: 0, iter: 10200, loss: 39.90406036376953
epoch: 0, iter: 10300, loss: 34.071014404296875
epoch: 0, iter: 10400, loss: 35.23554229736328
epoch: 0, iter: 10500, loss: 35.033878326416016
epoch: 0, iter: 10600, loss: 36.56634521484375
epoch: 0, iter: 10700, loss: 34.755027770996094
epoch: 0, iter: 10800, loss: 37.447967529296875
epoch: 0, iter: 10900, loss: 37.32883834838867
epoch: 0, iter: 11000, loss: 34.621700286865234
epoch: 0, iter: 11100, loss: 34.79033660888672
epoch: 0, iter: 11200, loss: 33.45790100097656
epoch: 0, iter: 11300, loss: 34.915672302246094
epoch: 0, iter: 11400, loss: 33.67906188964844
epoch: 0, iter: 11500, loss: 33.42378616333008
epoch: 0, iter: 11600, loss: 33.216270446777344
epoch: 0, iter: 11700, loss: 35.964393615722656
epoch: 0, iter: 11800, loss: 32.547569274902344
epoch: 0, iter: 11900, loss: 32.87192153930664
epoch: 0, iter: 12000, loss: 37.79120635986328
epoch: 0, iteration: 12000, simlex-999: SpearmanrResult(correlation=0.07427469122590927, pvalue=0.021568044209408773), men: SpearmanrResult(correlation=0.07554039135518772, pvalue=0.00011870202106880258), sim353: SpearmanrResult(correlation=0.003874488949244921, pvalue=0.9451327287240687), nearest to monster: ['monster', 'adoption', 'immediate', 'patent', 'sphere', 'execution', 'shell', 'nucleus', 'ghost', 'label']

epoch: 0, iter: 12100, loss: 33.59938430786133
epoch: 0, iter: 12200, loss: 32.594879150390625
epoch: 0, iter: 12300, loss: 32.42393493652344
epoch: 0, iter: 12400, loss: 32.8863410949707
epoch: 0, iter: 12500, loss: 39.303016662597656
epoch: 0, iter: 12600, loss: 33.103118896484375
epoch: 0, iter: 12700, loss: 36.31195068359375
epoch: 0, iter: 12800, loss: 33.8329963684082
epoch: 0, iter: 12900, loss: 32.499595642089844
epoch: 0, iter: 13000, loss: 33.224632263183594
epoch: 0, iter: 13100, loss: 33.931884765625
epoch: 0, iter: 13200, loss: 33.35892105102539
epoch: 0, iter: 13300, loss: 33.33966064453125
epoch: 0, iter: 13400, loss: 34.09075164794922
epoch: 0, iter: 13500, loss: 33.52397918701172
epoch: 0, iter: 13600, loss: 34.18444061279297
epoch: 0, iter: 13700, loss: 33.96720886230469
epoch: 0, iter: 13800, loss: 34.23271942138672
epoch: 0, iter: 13900, loss: 33.36094665527344
epoch: 0, iter: 14000, loss: 35.998287200927734
epoch: 0, iteration: 14000, simlex-999: SpearmanrResult(correlation=0.07498900438956249, pvalue=0.0203380930498303), men: SpearmanrResult(correlation=0.07885185599812983, pvalue=5.8687463983198815e-05), sim353: SpearmanrResult(correlation=0.019838726849964704, pvalue=0.7245257659604268), nearest to monster: ['monster', 'tale', 'patent', 'garden', 'outer', 'nucleus', 'logo', 'indians', 'fate', 'ghost']

epoch: 0, iter: 14100, loss: 32.86090087890625
epoch: 0, iter: 14200, loss: 32.27300262451172
epoch: 0, iter: 14300, loss: 32.97502136230469
epoch: 0, iter: 14400, loss: 33.18888473510742
epoch: 0, iter: 14500, loss: 33.709564208984375
epoch: 0, iter: 14600, loss: 33.725990295410156
epoch: 0, iter: 14700, loss: 34.124961853027344
epoch: 0, iter: 14800, loss: 34.69652557373047
epoch: 0, iter: 14900, loss: 36.399696350097656
epoch: 0, iter: 15000, loss: 32.656532287597656
epoch: 0, iter: 15100, loss: 33.403133392333984
epoch: 0, iter: 15200, loss: 32.11627960205078
epoch: 0, iter: 15300, loss: 32.489803314208984
epoch: 0, iter: 15400, loss: 32.96385192871094
epoch: 0, iter: 15500, loss: 33.85535430908203
epoch: 0, iter: 15600, loss: 33.443634033203125
epoch: 0, iter: 15700, loss: 32.89921569824219
epoch: 0, iter: 15800, loss: 31.661029815673828
epoch: 0, iter: 15900, loss: 32.627262115478516
epoch: 0, iter: 16000, loss: 32.10541534423828
epoch: 0, iteration: 16000, simlex-999: SpearmanrResult(correlation=0.0788889724045409, pvalue=0.014643454855412137), men: SpearmanrResult(correlation=0.08118046638145521, pvalue=3.517646407074078e-05), sim353: SpearmanrResult(correlation=0.03869824262332756, pvalue=0.49168668781560065), nearest to monster: ['monster', 'tale', 'patent', 'garden', 'logo', 'headquarters', 'floor', 'nucleus', 'hotel', 'outer']

epoch: 0, iter: 16100, loss: 32.40728759765625
epoch: 0, iter: 16200, loss: 32.153541564941406
epoch: 0, iter: 16300, loss: 32.54335021972656
epoch: 0, iter: 16400, loss: 33.81620788574219
epoch: 0, iter: 16500, loss: 33.6131591796875
epoch: 0, iter: 16600, loss: 33.012855529785156
epoch: 0, iter: 16700, loss: 32.9858512878418
epoch: 0, iter: 16800, loss: 32.339019775390625
epoch: 0, iter: 16900, loss: 33.2204475402832
epoch: 0, iter: 17000, loss: 32.71576690673828
epoch: 0, iter: 17100, loss: 33.55080795288086
epoch: 0, iter: 17200, loss: 32.817447662353516
epoch: 0, iter: 17300, loss: 34.78331756591797
epoch: 0, iter: 17400, loss: 32.013267517089844
epoch: 0, iter: 17500, loss: 32.31776428222656
epoch: 0, iter: 17600, loss: 32.81449508666992
epoch: 0, iter: 17700, loss: 32.663665771484375
epoch: 0, iter: 17800, loss: 32.64860534667969
epoch: 0, iter: 17900, loss: 32.25948715209961
epoch: 0, iter: 18000, loss: 33.899532318115234
epoch: 0, iteration: 18000, simlex-999: SpearmanrResult(correlation=0.08197570796707307, pvalue=0.01118359931439746), men: SpearmanrResult(correlation=0.08119437744439352, pvalue=3.5067625057299385e-05), sim353: SpearmanrResult(correlation=0.048031348197188906, pvalue=0.39330365782911914), nearest to monster: ['monster', 'tale', 'patent', 'household', 'dialogue', 'floor', 'sphere', 'mouse', 'fate', 'skin']

epoch: 0, iter: 18100, loss: 31.952678680419922
epoch: 0, iter: 18200, loss: 32.561737060546875
epoch: 0, iter: 18300, loss: 31.917354583740234
epoch: 0, iter: 18400, loss: 32.31993103027344
epoch: 0, iter: 18500, loss: 32.442169189453125
epoch: 0, iter: 18600, loss: 32.37964630126953
epoch: 0, iter: 18700, loss: 32.223846435546875
epoch: 0, iter: 18800, loss: 32.205589294433594
epoch: 0, iter: 18900, loss: 32.872222900390625
epoch: 0, iter: 19000, loss: 32.515403747558594
epoch: 0, iter: 19100, loss: 33.08296203613281
epoch: 0, iter: 19200, loss: 32.536170959472656
epoch: 0, iter: 19300, loss: 32.39844512939453
epoch: 0, iter: 19400, loss: 33.58967971801758
epoch: 0, iter: 19500, loss: 32.6730842590332
epoch: 0, iter: 19600, loss: 33.223388671875
epoch: 0, iter: 19700, loss: 32.08860397338867
epoch: 0, iter: 19800, loss: 31.78927993774414
epoch: 0, iter: 19900, loss: 31.92531967163086
epoch: 0, iter: 20000, loss: 32.14461898803711
epoch: 0, iteration: 20000, simlex-999: SpearmanrResult(correlation=0.08376406816249372, pvalue=0.00952959087521674), men: SpearmanrResult(correlation=0.08428805462978844, pvalue=1.7391127961421946e-05), sim353: SpearmanrResult(correlation=0.049551103193172526, pvalue=0.3784887673298559), nearest to monster: ['monster', 'patent', 'sword', 'household', 'dialogue', 'comprehensive', 'mouse', 'label', 'plain', 'tale']

epoch: 0, iter: 20100, loss: 32.788509368896484
epoch: 0, iter: 20200, loss: 31.743305206298828
epoch: 0, iter: 20300, loss: 32.98844909667969
epoch: 0, iter: 20400, loss: 32.939300537109375
epoch: 0, iter: 20500, loss: 32.22157669067383
epoch: 0, iter: 20600, loss: 32.10664367675781
epoch: 0, iter: 20700, loss: 32.317832946777344
epoch: 0, iter: 20800, loss: 32.22321701049805
epoch: 0, iter: 20900, loss: 32.078826904296875
epoch: 0, iter: 21000, loss: 32.00135803222656
epoch: 0, iter: 21100, loss: 32.2218017578125
epoch: 0, iter: 21200, loss: 32.36552047729492
epoch: 0, iter: 21300, loss: 32.28803253173828
epoch: 0, iter: 21400, loss: 32.49916076660156
epoch: 0, iter: 21500, loss: 31.330402374267578
epoch: 0, iter: 21600, loss: 32.153507232666016
epoch: 0, iter: 21700, loss: 32.27666473388672
epoch: 0, iter: 21800, loss: 31.28035545349121
epoch: 0, iter: 21900, loss: 31.78491973876953
epoch: 0, iter: 22000, loss: 32.09901428222656
epoch: 0, iteration: 22000, simlex-999: SpearmanrResult(correlation=0.0841933673566154, pvalue=0.009166514672039517), men: SpearmanrResult(correlation=0.08568243547516359, pvalue=1.2577781665179613e-05), sim353: SpearmanrResult(correlation=0.05233237611768227, pvalue=0.3522765894341572), nearest to monster: ['monster', 'sword', 'hero', 'ghost', 'patent', 'tale', 'comprehensive', 'plain', 'household', 'goddess']

epoch: 0, iter: 22100, loss: 32.55064392089844
epoch: 0, iter: 22200, loss: 32.269989013671875
epoch: 0, iter: 22300, loss: 31.861957550048828
epoch: 0, iter: 22400, loss: 35.57160949707031
epoch: 0, iter: 22500, loss: 31.28049087524414
epoch: 0, iter: 22600, loss: 32.447288513183594
epoch: 0, iter: 22700, loss: 31.807647705078125
epoch: 0, iter: 22800, loss: 31.540283203125
epoch: 0, iter: 22900, loss: 31.646018981933594
epoch: 0, iter: 23000, loss: 32.140228271484375
epoch: 0, iter: 23100, loss: 31.19212532043457
epoch: 0, iter: 23200, loss: 32.096595764160156
epoch: 0, iter: 23300, loss: 32.60624313354492
epoch: 0, iter: 23400, loss: 31.942745208740234
epoch: 0, iter: 23500, loss: 32.21788787841797
epoch: 0, iter: 23600, loss: 32.34299087524414
epoch: 0, iter: 23700, loss: 31.90642547607422
epoch: 0, iter: 23800, loss: 31.761348724365234
epoch: 0, iter: 23900, loss: 32.32670211791992
epoch: 0, iter: 24000, loss: 32.27470397949219
epoch: 0, iteration: 24000, simlex-999: SpearmanrResult(correlation=0.08547535475215376, pvalue=0.008154549896277891), men: SpearmanrResult(correlation=0.08635481027650124, pvalue=1.073940217602237e-05), sim353: SpearmanrResult(correlation=0.05715118428542604, pvalue=0.309644216967956), nearest to monster: ['monster', 'sword', 'hero', 'ghost', 'plain', 'household', 'situated', 'brand', 'torah', 'mouse']

epoch: 0, iter: 24100, loss: 31.711109161376953
epoch: 0, iter: 24200, loss: 31.729236602783203
epoch: 0, iter: 24300, loss: 31.751216888427734
epoch: 0, iter: 24400, loss: 31.54802131652832
epoch: 0, iter: 24500, loss: 31.819448471069336
epoch: 0, iter: 24600, loss: 31.87582778930664
epoch: 0, iter: 24700, loss: 32.44230651855469
epoch: 0, iter: 24800, loss: 32.13909149169922
epoch: 0, iter: 24900, loss: 31.6838321685791
epoch: 0, iter: 25000, loss: 32.01523208618164
epoch: 0, iter: 25100, loss: 31.727489471435547
epoch: 0, iter: 25200, loss: 32.378543853759766
epoch: 0, iter: 25300, loss: 32.155052185058594
epoch: 0, iter: 25400, loss: 32.30049514770508
epoch: 0, iter: 25500, loss: 32.10628128051758
epoch: 0, iter: 25600, loss: 32.01287841796875
epoch: 0, iter: 25700, loss: 32.22496032714844
epoch: 0, iter: 25800, loss: 32.15202331542969
epoch: 0, iter: 25900, loss: 32.43567657470703
epoch: 0, iter: 26000, loss: 31.745975494384766
epoch: 0, iteration: 26000, simlex-999: SpearmanrResult(correlation=0.08715629365703002, pvalue=0.0069793574982666565), men: SpearmanrResult(correlation=0.08749437789759629, pvalue=8.194697761171436e-06), sim353: SpearmanrResult(correlation=0.05971657549964074, pvalue=0.28839311254438554), nearest to monster: ['monster', 'sword', 'household', 'hero', 'tale', 'priest', 'label', 'plain', 'mouse', 'ghost']

epoch: 0, iter: 26100, loss: 32.09526824951172
epoch: 0, iter: 26200, loss: 31.927221298217773
epoch: 0, iter: 26300, loss: 31.239913940429688
epoch: 0, iter: 26400, loss: 31.676891326904297
epoch: 0, iter: 26500, loss: 31.83584213256836
epoch: 0, iter: 26600, loss: 32.34405517578125
epoch: 0, iter: 26700, loss: 31.836318969726562
epoch: 0, iter: 26800, loss: 31.805145263671875
epoch: 0, iter: 26900, loss: 31.517250061035156
epoch: 0, iter: 27000, loss: 32.060646057128906
epoch: 0, iter: 27100, loss: 31.427961349487305
epoch: 0, iter: 27200, loss: 32.71056365966797
epoch: 0, iter: 27300, loss: 32.101768493652344
epoch: 0, iter: 27400, loss: 31.706729888916016
epoch: 0, iter: 27500, loss: 31.794944763183594
epoch: 0, iter: 27600, loss: 31.043569564819336
epoch: 0, iter: 27700, loss: 31.815420150756836
epoch: 0, iter: 27800, loss: 31.480987548828125
epoch: 0, iter: 27900, loss: 32.0
epoch: 0, iter: 28000, loss: 31.647756576538086
epoch: 0, iteration: 28000, simlex-999: SpearmanrResult(correlation=0.0879225360679704, pvalue=0.006495932231970623), men: SpearmanrResult(correlation=0.08939464976521133, pvalue=5.181905435780726e-06), sim353: SpearmanrResult(correlation=0.06028361484068362, pvalue=0.28383120456458), nearest to monster: ['monster', 'sword', 'plain', 'mouse', 'tale', 'hero', 'brand', 'patent', 'tail', 'ghost']

epoch: 0, iter: 28100, loss: 31.732290267944336
epoch: 0, iter: 28200, loss: 31.56043243408203
epoch: 0, iter: 28300, loss: 32.17532730102539
epoch: 0, iter: 28400, loss: 32.34858322143555
epoch: 0, iter: 28500, loss: 31.321521759033203
epoch: 0, iter: 28600, loss: 31.24187469482422
epoch: 0, iter: 28700, loss: 31.808574676513672
epoch: 0, iter: 28800, loss: 31.126705169677734
epoch: 0, iter: 28900, loss: 32.27989959716797
epoch: 0, iter: 29000, loss: 31.510923385620117
epoch: 0, iter: 29100, loss: 32.18346405029297
epoch: 0, iter: 29200, loss: 31.19722557067871
epoch: 0, iter: 29300, loss: 31.348796844482422
epoch: 0, iter: 29400, loss: 32.03580856323242
epoch: 0, iter: 29500, loss: 31.560871124267578
epoch: 0, iter: 29600, loss: 32.1707763671875
epoch: 0, iter: 29700, loss: 31.646257400512695
epoch: 0, iter: 29800, loss: 32.089317321777344
epoch: 0, iter: 29900, loss: 31.21417808532715
epoch: 0, iter: 30000, loss: 31.596721649169922
epoch: 0, iteration: 30000, simlex-999: SpearmanrResult(correlation=0.0908617796694403, pvalue=0.00490825911563686), men: SpearmanrResult(correlation=0.09006953525508496, pvalue=4.393754176783815e-06), sim353: SpearmanrResult(correlation=0.06781615126644898, pvalue=0.22783225512951796), nearest to monster: ['monster', 'hero', 'sword', 'mouse', 'nickname', 'tale', 'plain', 'ghost', 'expedition', 'tube']

epoch: 0, iter: 30100, loss: 31.1719970703125
epoch: 0, iter: 30200, loss: 31.563777923583984
epoch: 0, iter: 30300, loss: 31.362476348876953
epoch: 0, iter: 30400, loss: 31.93914222717285
epoch: 0, iter: 30500, loss: 31.46084213256836
epoch: 0, iter: 30600, loss: 31.61031723022461
epoch: 0, iter: 30700, loss: 32.19886779785156
epoch: 0, iter: 30800, loss: 31.53145980834961
epoch: 0, iter: 30900, loss: 32.54494094848633
epoch: 0, iter: 31000, loss: 31.622350692749023
epoch: 0, iter: 31100, loss: 31.624622344970703
epoch: 0, iter: 31200, loss: 32.21925354003906
epoch: 0, iter: 31300, loss: 31.355022430419922
epoch: 0, iter: 31400, loss: 31.890806198120117
epoch: 0, iter: 31500, loss: 31.63449478149414
epoch: 0, iter: 31600, loss: 31.105436325073242
epoch: 0, iter: 31700, loss: 31.645238876342773
epoch: 0, iter: 31800, loss: 31.808307647705078
epoch: 0, iter: 31900, loss: 31.988243103027344
epoch: 0, iter: 32000, loss: 31.83148193359375
epoch: 0, iteration: 32000, simlex-999: SpearmanrResult(correlation=0.0910634737532756, pvalue=0.004813389051845152), men: SpearmanrResult(correlation=0.09222228979601282, pvalue=2.5756952028504964e-06), sim353: SpearmanrResult(correlation=0.07123238137344272, pvalue=0.20520429313982647), nearest to monster: ['monster', 'hero', 'nickname', 'sword', 'tale', 'plain', 'mouse', 'ghost', 'tail', 'tube']

epoch: 0, iter: 32100, loss: 31.311588287353516
epoch: 0, iter: 32200, loss: 31.257244110107422
epoch: 0, iter: 32300, loss: 31.649892807006836
epoch: 0, iter: 32400, loss: 31.635969161987305
epoch: 0, iter: 32500, loss: 31.34613037109375
epoch: 0, iter: 32600, loss: 31.666229248046875
epoch: 0, iter: 32700, loss: 31.63262176513672
epoch: 0, iter: 32800, loss: 31.727909088134766
epoch: 0, iter: 32900, loss: 32.014007568359375
epoch: 0, iter: 33000, loss: 31.64935302734375
epoch: 0, iter: 33100, loss: 31.75027084350586
epoch: 0, iter: 33200, loss: 30.913625717163086
epoch: 0, iter: 33300, loss: 32.485591888427734
epoch: 0, iter: 33400, loss: 30.946617126464844
epoch: 0, iter: 33500, loss: 31.906150817871094
epoch: 0, iter: 33600, loss: 31.456090927124023
epoch: 0, iter: 33700, loss: 31.70574188232422
epoch: 0, iter: 33800, loss: 31.611658096313477
epoch: 0, iter: 33900, loss: 31.901599884033203
epoch: 0, iter: 34000, loss: 30.904211044311523
epoch: 0, iteration: 34000, simlex-999: SpearmanrResult(correlation=0.09100297719676911, pvalue=0.004841669162232116), men: SpearmanrResult(correlation=0.09320434671047619, pvalue=2.0108242547194325e-06), sim353: SpearmanrResult(correlation=0.07643014870593084, pvalue=0.1739669852121724), nearest to monster: ['monster', 'nickname', 'tale', 'mouse', 'brand', 'hero', 'plain', 'partner', 'owner', 'cave']

epoch: 0, iter: 34100, loss: 30.875843048095703
epoch: 0, iter: 34200, loss: 31.53815460205078
epoch: 0, iter: 34300, loss: 31.3868465423584
epoch: 0, iter: 34400, loss: 31.618576049804688
epoch: 0, iter: 34500, loss: 31.38482666015625
epoch: 0, iter: 34600, loss: 31.517066955566406
epoch: 0, iter: 34700, loss: 31.297931671142578
epoch: 0, iter: 34800, loss: 31.131715774536133
epoch: 0, iter: 34900, loss: 31.34206199645996
epoch: 0, iter: 35000, loss: 31.198501586914062
epoch: 0, iter: 35100, loss: 31.92325782775879
epoch: 0, iter: 35200, loss: 31.495628356933594
epoch: 0, iter: 35300, loss: 31.19044303894043
epoch: 0, iter: 35400, loss: 31.896709442138672
epoch: 0, iter: 35500, loss: 31.638015747070312
epoch: 0, iter: 35600, loss: 31.722248077392578
epoch: 0, iter: 35700, loss: 31.750402450561523
epoch: 0, iter: 35800, loss: 31.107473373413086
epoch: 0, iter: 35900, loss: 31.830018997192383
epoch: 0, iter: 36000, loss: 31.638286590576172
epoch: 0, iteration: 36000, simlex-999: SpearmanrResult(correlation=0.09287122747290566, pvalue=0.004034242774511441), men: SpearmanrResult(correlation=0.09638454243178861, pvalue=8.867706115523595e-07), sim353: SpearmanrResult(correlation=0.08196414667104787, pvalue=0.14474986358858538), nearest to monster: ['monster', 'nickname', 'plain', 'sword', 'tail', 'mouse', 'brand', 'hero', 'tale', 'shell']

epoch: 0, iter: 36100, loss: 30.960386276245117
epoch: 0, iter: 36200, loss: 31.629940032958984
epoch: 0, iter: 36300, loss: 31.541032791137695
epoch: 0, iter: 36400, loss: 31.05801773071289
epoch: 0, iter: 36500, loss: 31.969802856445312
epoch: 0, iter: 36600, loss: 31.290489196777344
epoch: 0, iter: 36700, loss: 31.409465789794922
epoch: 0, iter: 36800, loss: 31.444076538085938
epoch: 0, iter: 36900, loss: 31.494474411010742
epoch: 0, iter: 37000, loss: 31.12554931640625
epoch: 0, iter: 37100, loss: 31.744049072265625
epoch: 0, iter: 37200, loss: 31.608917236328125
epoch: 0, iter: 37300, loss: 31.441722869873047
epoch: 0, iter: 37400, loss: 31.544227600097656
epoch: 0, iter: 37500, loss: 31.359806060791016
epoch: 0, iter: 37600, loss: 31.130847930908203
epoch: 0, iter: 37700, loss: 32.14916229248047
epoch: 0, iter: 37800, loss: 31.148212432861328
epoch: 0, iter: 37900, loss: 31.835248947143555
epoch: 0, iter: 38000, loss: 31.421974182128906
epoch: 0, iteration: 38000, simlex-999: SpearmanrResult(correlation=0.09401565185194706, pvalue=0.003602024110356835), men: SpearmanrResult(correlation=0.09723017395213002, pvalue=7.101718335843492e-07), sim353: SpearmanrResult(correlation=0.08744795260499, pvalue=0.1196457667795805), nearest to monster: ['monster', 'nickname', 'plain', 'sword', 'tail', 'hero', 'shell', 'brand', 'mouse', 'cave']

epoch: 0, iter: 38100, loss: 30.881635665893555
epoch: 0, iter: 38200, loss: 31.16852569580078
epoch: 0, iter: 38300, loss: 31.63935089111328
epoch: 0, iter: 38400, loss: 31.25921058654785
epoch: 0, iter: 38500, loss: 31.57360076904297
epoch: 0, iter: 38600, loss: 31.1456298828125
epoch: 0, iter: 38700, loss: 31.269453048706055
epoch: 0, iter: 38800, loss: 31.55490493774414
epoch: 0, iter: 38900, loss: 31.626995086669922
epoch: 0, iter: 39000, loss: 31.255146026611328
epoch: 0, iter: 39100, loss: 31.211166381835938
epoch: 0, iter: 39200, loss: 31.450740814208984
epoch: 0, iter: 39300, loss: 31.785430908203125
epoch: 0, iter: 39400, loss: 30.558988571166992
epoch: 0, iter: 39500, loss: 31.32469940185547
epoch: 0, iter: 39600, loss: 31.5628604888916
epoch: 0, iter: 39700, loss: 31.271900177001953
epoch: 0, iter: 39800, loss: 31.499229431152344
epoch: 0, iter: 39900, loss: 31.45954704284668
epoch: 0, iter: 40000, loss: 30.844253540039062
epoch: 0, iteration: 40000, simlex-999: SpearmanrResult(correlation=0.09659229076909075, pvalue=0.0027787052714036363), men: SpearmanrResult(correlation=0.09859835382112378, pvalue=4.938796863215718e-07), sim353: SpearmanrResult(correlation=0.09091941260502377, pvalue=0.10559925075777613), nearest to monster: ['monster', 'nickname', 'plain', 'hero', 'cave', 'sword', 'tail', 'owner', 'dialogue', 'mouse']

epoch: 0, iter: 40100, loss: 31.289958953857422
epoch: 0, iter: 40200, loss: 31.427631378173828
epoch: 0, iter: 40300, loss: 30.93175506591797
epoch: 0, iter: 40400, loss: 31.097423553466797
epoch: 0, iter: 40500, loss: 31.367881774902344
epoch: 0, iter: 40600, loss: 30.997957229614258
epoch: 0, iter: 40700, loss: 31.378498077392578
epoch: 0, iter: 40800, loss: 31.591278076171875
epoch: 0, iter: 40900, loss: 31.236934661865234
epoch: 0, iter: 41000, loss: 31.594310760498047
epoch: 0, iter: 41100, loss: 31.448932647705078
epoch: 0, iter: 41200, loss: 30.75921058654785
epoch: 0, iter: 41300, loss: 31.807411193847656
epoch: 0, iter: 41400, loss: 30.96005630493164
epoch: 0, iter: 41500, loss: 31.805885314941406
epoch: 0, iter: 41600, loss: 31.190258026123047
epoch: 0, iter: 41700, loss: 31.110252380371094
epoch: 0, iter: 41800, loss: 31.04319190979004
epoch: 0, iter: 41900, loss: 30.97702407836914
epoch: 0, iter: 42000, loss: 31.0760440826416
epoch: 0, iteration: 42000, simlex-999: SpearmanrResult(correlation=0.09849883639358918, pvalue=0.0022842954860782523), men: SpearmanrResult(correlation=0.09878607981201826, pvalue=4.696928075901211e-07), sim353: SpearmanrResult(correlation=0.09607407823044349, pvalue=0.08718116473821737), nearest to monster: ['monster', 'cave', 'plain', 'mouse', 'nickname', 'diamond', 'dialogue', 'partner', 'hero', 'signature']

epoch: 0, iter: 42100, loss: 31.264083862304688
epoch: 0, iter: 42200, loss: 31.662830352783203
epoch: 0, iter: 42300, loss: 30.693662643432617
epoch: 0, iter: 42400, loss: 31.405860900878906
epoch: 0, iter: 42500, loss: 30.938379287719727
epoch: 0, iter: 42600, loss: 31.08720588684082
epoch: 0, iter: 42700, loss: 31.5560302734375
epoch: 0, iter: 42800, loss: 31.49104881286621
epoch: 0, iter: 42900, loss: 31.75652503967285
epoch: 0, iter: 43000, loss: 31.436534881591797
epoch: 0, iter: 43100, loss: 31.30294418334961
epoch: 0, iter: 43200, loss: 30.177589416503906
epoch: 0, iter: 43300, loss: 31.117063522338867
epoch: 0, iter: 43400, loss: 30.985565185546875
epoch: 0, iter: 43500, loss: 30.83687973022461
epoch: 0, iter: 43600, loss: 31.235471725463867
epoch: 0, iter: 43700, loss: 31.702655792236328
epoch: 0, iter: 43800, loss: 31.2994441986084
epoch: 0, iter: 43900, loss: 30.892574310302734
epoch: 0, iter: 44000, loss: 31.143707275390625
epoch: 0, iteration: 44000, simlex-999: SpearmanrResult(correlation=0.1008180671120631, pvalue=0.001791807298558225), men: SpearmanrResult(correlation=0.09959304604435494, pvalue=3.781135368008198e-07), sim353: SpearmanrResult(correlation=0.10352488934150769, pvalue=0.06521224014872654), nearest to monster: ['monster', 'mouse', 'plain', 'nickname', 'cave', 'sword', 'boat', 'dialogue', 'partner', 'signature']

epoch: 0, iter: 44100, loss: 30.712566375732422
epoch: 0, iter: 44200, loss: 31.178279876708984
epoch: 0, iter: 44300, loss: 31.13910484313965
epoch: 0, iter: 44400, loss: 31.227054595947266
epoch: 0, iter: 44500, loss: 31.095703125
epoch: 0, iter: 44600, loss: 31.12378692626953
epoch: 0, iter: 44700, loss: 31.701740264892578
epoch: 0, iter: 44800, loss: 30.913339614868164
epoch: 0, iter: 44900, loss: 31.539695739746094
epoch: 0, iter: 45000, loss: 31.188980102539062
epoch: 0, iter: 45100, loss: 30.845016479492188
epoch: 0, iter: 45200, loss: 30.882841110229492
epoch: 0, iter: 45300, loss: 31.02661895751953
epoch: 0, iter: 45400, loss: 31.336511611938477
epoch: 0, iter: 45500, loss: 31.420623779296875
epoch: 0, iter: 45600, loss: 31.25517463684082
epoch: 0, iter: 45700, loss: 31.28260040283203
epoch: 0, iter: 45800, loss: 31.164663314819336
epoch: 0, iter: 45900, loss: 31.538354873657227
epoch: 0, iter: 46000, loss: 30.74416732788086
epoch: 0, iteration: 46000, simlex-999: SpearmanrResult(correlation=0.10155071688717769, pvalue=0.0016577893244519035), men: SpearmanrResult(correlation=0.10163250557010925, pvalue=2.1692539986465238e-07), sim353: SpearmanrResult(correlation=0.10604008852454751, pvalue=0.058912219453194116), nearest to monster: ['monster', 'plain', 'nickname', 'sword', 'parent', 'mouse', 'dialogue', 'blade', 'boat', 'cave']

epoch: 0, iter: 46100, loss: 31.790225982666016
epoch: 0, iter: 46200, loss: 31.57557487487793
epoch: 0, iter: 46300, loss: 31.457191467285156
epoch: 0, iter: 46400, loss: 31.26146697998047
epoch: 0, iter: 46500, loss: 31.269039154052734
epoch: 0, iter: 46600, loss: 31.480712890625
epoch: 0, iter: 46700, loss: 31.428335189819336
epoch: 0, iter: 46800, loss: 30.886512756347656
epoch: 0, iter: 46900, loss: 31.599918365478516
epoch: 0, iter: 47000, loss: 31.132366180419922
epoch: 0, iter: 47100, loss: 30.962696075439453
epoch: 0, iter: 47200, loss: 31.67426300048828
epoch: 0, iter: 47300, loss: 30.73318862915039
epoch: 0, iter: 47400, loss: 31.673181533813477
epoch: 0, iter: 47500, loss: 31.30075454711914
epoch: 0, iter: 47600, loss: 31.427719116210938
epoch: 0, iter: 47700, loss: 31.111129760742188
epoch: 0, iter: 47800, loss: 31.14962387084961
epoch: 0, iter: 47900, loss: 31.174724578857422
epoch: 0, iter: 48000, loss: 31.114784240722656
epoch: 0, iteration: 48000, simlex-999: SpearmanrResult(correlation=0.10335720444738267, pvalue=0.0013657280173897167), men: SpearmanrResult(correlation=0.10165177484469035, pvalue=2.157784847483254e-07), sim353: SpearmanrResult(correlation=0.11041761731040038, pvalue=0.049149368437484346), nearest to monster: ['monster', 'plain', 'nickname', 'parent', 'sword', 'blade', 'tail', 'dialogue', 'cave', 'boat']

epoch: 0, iter: 48100, loss: 31.263320922851562
epoch: 0, iter: 48200, loss: 31.267719268798828
epoch: 0, iter: 48300, loss: 31.259817123413086
epoch: 0, iter: 48400, loss: 30.922761917114258
epoch: 0, iter: 48500, loss: 31.48914909362793
epoch: 0, iter: 48600, loss: 31.45376205444336
epoch: 0, iter: 48700, loss: 30.948339462280273
epoch: 0, iter: 48800, loss: 30.842824935913086
epoch: 0, iter: 48900, loss: 30.931697845458984
epoch: 0, iter: 49000, loss: 31.468204498291016
epoch: 0, iter: 49100, loss: 31.04726791381836
epoch: 0, iter: 49200, loss: 31.148698806762695
epoch: 0, iter: 49300, loss: 31.295198440551758
epoch: 0, iter: 49400, loss: 31.415983200073242
epoch: 0, iter: 49500, loss: 31.53121566772461
epoch: 0, iter: 49600, loss: 30.391773223876953
epoch: 0, iter: 49700, loss: 31.365924835205078
epoch: 0, iter: 49800, loss: 30.920448303222656
epoch: 0, iter: 49900, loss: 30.881540298461914
epoch: 0, iter: 50000, loss: 31.272510528564453
epoch: 0, iteration: 50000, simlex-999: SpearmanrResult(correlation=0.10413335271622073, pvalue=0.0012554545146236879), men: SpearmanrResult(correlation=0.10361287469529604, pvalue=1.251734153196469e-07), sim353: SpearmanrResult(correlation=0.11252176428274015, pvalue=0.04496085066226139), nearest to monster: ['monster', 'parent', 'sword', 'nickname', 'boat', 'plain', 'tail', 'leg', 'mouse', 'blade']

epoch: 0, iter: 50100, loss: 31.378141403198242
epoch: 0, iter: 50200, loss: 30.816102981567383
epoch: 0, iter: 50300, loss: 30.845239639282227
epoch: 0, iter: 50400, loss: 30.991004943847656
epoch: 0, iter: 50500, loss: 30.891719818115234
epoch: 0, iter: 50600, loss: 31.482940673828125
epoch: 0, iter: 50700, loss: 31.31090545654297
epoch: 0, iter: 50800, loss: 31.34703826904297
epoch: 0, iter: 50900, loss: 31.271032333374023
epoch: 0, iter: 51000, loss: 31.262798309326172
epoch: 0, iter: 51100, loss: 31.295764923095703
epoch: 0, iter: 51200, loss: 31.204692840576172
epoch: 0, iter: 51300, loss: 31.768779754638672
epoch: 0, iter: 51400, loss: 30.988128662109375
epoch: 0, iter: 51500, loss: 31.494434356689453
epoch: 0, iter: 51600, loss: 31.034160614013672
epoch: 0, iter: 51700, loss: 31.57693099975586
epoch: 0, iter: 51800, loss: 31.073469161987305
epoch: 0, iter: 51900, loss: 30.947439193725586
epoch: 0, iter: 52000, loss: 31.44693374633789
epoch: 0, iteration: 52000, simlex-999: SpearmanrResult(correlation=0.10512161472015334, pvalue=0.0011269247727856127), men: SpearmanrResult(correlation=0.10482327567400625, pvalue=8.899831939238548e-08), sim353: SpearmanrResult(correlation=0.11557228293582009, pvalue=0.03942341315579966), nearest to monster: ['monster', 'parent', 'plain', 'nickname', 'tail', 'blade', 'leg', 'sword', 'mouse', 'signature']

epoch: 0, iter: 52100, loss: 31.460336685180664
epoch: 0, iter: 52200, loss: 31.668170928955078
epoch: 0, iter: 52300, loss: 30.479820251464844
epoch: 0, iter: 52400, loss: 31.74282455444336
epoch: 0, iter: 52500, loss: 30.803956985473633
epoch: 0, iter: 52600, loss: 30.81981658935547
epoch: 0, iter: 52700, loss: 31.19491958618164
epoch: 0, iter: 52800, loss: 31.08262825012207
epoch: 0, iter: 52900, loss: 31.718488693237305
epoch: 0, iter: 53000, loss: 30.93524932861328
epoch: 0, iter: 53100, loss: 31.013084411621094
epoch: 0, iter: 53200, loss: 30.91759490966797
epoch: 0, iter: 53300, loss: 31.814682006835938
epoch: 0, iter: 53400, loss: 30.962512969970703
epoch: 0, iter: 53500, loss: 30.939342498779297
epoch: 0, iter: 53600, loss: 31.67828369140625
epoch: 0, iter: 53700, loss: 31.302024841308594
epoch: 0, iter: 53800, loss: 30.955366134643555
epoch: 0, iter: 53900, loss: 31.510623931884766
epoch: 0, iter: 54000, loss: 30.676362991333008
epoch: 0, iteration: 54000, simlex-999: SpearmanrResult(correlation=0.10549625057039985, pvalue=0.0010814608115210568), men: SpearmanrResult(correlation=0.10721055714110006, pvalue=4.4913037691946484e-08), sim353: SpearmanrResult(correlation=0.11769079714384932, pvalue=0.03592648928681193), nearest to monster: ['monster', 'cave', 'plain', 'tail', 'nickname', 'leg', 'parent', 'blade', 'ghost', 'sword']

epoch: 0, iter: 54100, loss: 30.862953186035156
epoch: 0, iter: 54200, loss: 31.025741577148438
epoch: 0, iter: 54300, loss: 31.63794708251953
epoch: 0, iter: 54400, loss: 31.128381729125977
epoch: 0, iter: 54500, loss: 31.111047744750977
epoch: 0, iter: 54600, loss: 31.013673782348633
epoch: 0, iter: 54700, loss: 31.087692260742188
epoch: 0, iter: 54800, loss: 31.236892700195312
epoch: 0, iter: 54900, loss: 31.256492614746094
epoch: 0, iter: 55000, loss: 31.33741569519043
epoch: 0, iter: 55100, loss: 30.920692443847656
epoch: 0, iter: 55200, loss: 30.418161392211914
epoch: 0, iter: 55300, loss: 31.029638290405273
epoch: 0, iter: 55400, loss: 30.816083908081055
epoch: 0, iter: 55500, loss: 31.28571891784668
epoch: 0, iter: 55600, loss: 30.913631439208984
epoch: 0, iter: 55700, loss: 31.216411590576172
epoch: 0, iter: 55800, loss: 30.906490325927734
epoch: 0, iter: 55900, loss: 31.138019561767578
epoch: 0, iter: 56000, loss: 31.294254302978516
epoch: 0, iteration: 56000, simlex-999: SpearmanrResult(correlation=0.10747042881452093, pvalue=0.000868628570627656), men: SpearmanrResult(correlation=0.10889856420740306, pvalue=2.7445360530151176e-08), sim353: SpearmanrResult(correlation=0.12059633732769914, pvalue=0.03156119729084597), nearest to monster: ['monster', 'leg', 'tail', 'plain', 'nickname', 'sword', 'cave', 'parent', 'signature', 'blade']

epoch: 0, iter: 56100, loss: 31.756465911865234
epoch: 0, iter: 56200, loss: 30.841522216796875
epoch: 0, iter: 56300, loss: 31.101408004760742
epoch: 0, iter: 56400, loss: 30.875770568847656
epoch: 0, iter: 56500, loss: 31.620193481445312
epoch: 0, iter: 56600, loss: 31.299055099487305
epoch: 0, iter: 56700, loss: 31.310087203979492
epoch: 0, iter: 56800, loss: 31.34184455871582
epoch: 0, iter: 56900, loss: 31.240936279296875
epoch: 0, iter: 57000, loss: 30.90418815612793
epoch: 0, iter: 57100, loss: 31.257062911987305
epoch: 0, iter: 57200, loss: 31.695873260498047
epoch: 0, iter: 57300, loss: 30.94247055053711
epoch: 0, iter: 57400, loss: 30.684371948242188
epoch: 0, iter: 57500, loss: 31.504837036132812
epoch: 0, iter: 57600, loss: 31.262527465820312
epoch: 0, iter: 57700, loss: 31.38228988647461
epoch: 0, iter: 57800, loss: 31.27547836303711
epoch: 0, iter: 57900, loss: 30.56299591064453
epoch: 0, iter: 58000, loss: 30.96505355834961
epoch: 0, iteration: 58000, simlex-999: SpearmanrResult(correlation=0.10726587013548153, pvalue=0.0008887284442590619), men: SpearmanrResult(correlation=0.1093378019236274, pvalue=2.411455598304873e-08), sim353: SpearmanrResult(correlation=0.12085774638225694, pvalue=0.031191656645013426), nearest to monster: ['monster', 'tail', 'mouse', 'plain', 'blade', 'cave', 'signature', 'dubbed', 'angel', 'leg']

epoch: 0, iter: 58100, loss: 30.774898529052734
epoch: 0, iter: 58200, loss: 31.404376983642578
epoch: 0, iter: 58300, loss: 31.18744659423828
epoch: 0, iter: 58400, loss: 30.777774810791016
epoch: 0, iter: 58500, loss: 31.30267333984375
epoch: 0, iter: 58600, loss: 31.17416763305664
epoch: 0, iter: 58700, loss: 30.96724510192871
epoch: 0, iter: 58800, loss: 31.406417846679688
epoch: 0, iter: 58900, loss: 30.588558197021484
epoch: 0, iter: 59000, loss: 30.476577758789062
epoch: 0, iter: 59100, loss: 30.78055763244629
epoch: 0, iter: 59200, loss: 31.018653869628906
epoch: 0, iter: 59300, loss: 31.332136154174805
epoch: 0, iter: 59400, loss: 31.59677505493164
epoch: 0, iter: 59500, loss: 31.528217315673828
epoch: 0, iter: 59600, loss: 30.44378662109375
epoch: 0, iter: 59700, loss: 30.718303680419922
epoch: 0, iter: 59800, loss: 30.775535583496094
epoch: 0, iter: 59900, loss: 31.164199829101562
epoch: 0, iter: 60000, loss: 31.144628524780273
epoch: 0, iteration: 60000, simlex-999: SpearmanrResult(correlation=0.10787079798749166, pvalue=0.0008305009335772662), men: SpearmanrResult(correlation=0.11117447271855486, pvalue=1.3962055373097627e-08), sim353: SpearmanrResult(correlation=0.12349889211192522, pvalue=0.027660751055994845), nearest to monster: ['monster', 'tail', 'signature', 'blade', 'angel', 'dubbed', 'mouse', 'pole', 'owner', 'plain']

epoch: 0, iter: 60100, loss: 31.20758056640625
epoch: 0, iter: 60200, loss: 30.711570739746094
epoch: 0, iter: 60300, loss: 30.836360931396484
epoch: 0, iter: 60400, loss: 30.35114097595215
epoch: 0, iter: 60500, loss: 30.544240951538086
epoch: 0, iter: 60600, loss: 31.264543533325195
epoch: 0, iter: 60700, loss: 31.218517303466797
epoch: 0, iter: 60800, loss: 31.23360824584961
epoch: 0, iter: 60900, loss: 30.85096549987793
epoch: 0, iter: 61000, loss: 30.768386840820312
epoch: 0, iter: 61100, loss: 31.50748634338379
epoch: 0, iter: 61200, loss: 30.46345329284668
epoch: 0, iter: 61300, loss: 30.543607711791992
epoch: 0, iter: 61400, loss: 30.628982543945312
epoch: 0, iter: 61500, loss: 31.45627784729004
epoch: 0, iter: 61600, loss: 31.070459365844727
epoch: 0, iter: 61700, loss: 30.569217681884766
epoch: 0, iter: 61800, loss: 30.83639907836914
epoch: 0, iter: 61900, loss: 31.005922317504883
epoch: 0, iter: 62000, loss: 31.41488265991211
epoch: 0, iteration: 62000, simlex-999: SpearmanrResult(correlation=0.11119875283206068, pvalue=0.0005685786512505508), men: SpearmanrResult(correlation=0.11318488733549789, pvalue=7.599257092187759e-09), sim353: SpearmanrResult(correlation=0.12779805415765372, pvalue=0.022646548827240445), nearest to monster: ['monster', 'tail', 'blade', 'signature', 'plain', 'mouse', 'pole', 'boat', 'owner', 'leg']

epoch: 0, iter: 62100, loss: 31.053007125854492
epoch: 0, iter: 62200, loss: 30.765029907226562
epoch: 0, iter: 62300, loss: 31.114418029785156
epoch: 0, iter: 62400, loss: 30.98143768310547
epoch: 0, iter: 62500, loss: 31.071922302246094
epoch: 0, iter: 62600, loss: 31.17368507385254
epoch: 0, iter: 62700, loss: 31.177242279052734
epoch: 0, iter: 62800, loss: 31.408926010131836
epoch: 0, iter: 62900, loss: 30.88961410522461
epoch: 0, iter: 63000, loss: 30.848337173461914
epoch: 0, iter: 63100, loss: 30.798885345458984
epoch: 0, iter: 63200, loss: 30.96042251586914
epoch: 0, iter: 63300, loss: 30.656030654907227
epoch: 0, iter: 63400, loss: 31.166887283325195
epoch: 0, iter: 63500, loss: 30.926340103149414
epoch: 0, iter: 63600, loss: 31.11106300354004
epoch: 0, iter: 63700, loss: 31.001605987548828
epoch: 0, iter: 63800, loss: 30.872831344604492
epoch: 0, iter: 63900, loss: 31.2712345123291
epoch: 0, iter: 64000, loss: 31.084636688232422
epoch: 0, iteration: 64000, simlex-999: SpearmanrResult(correlation=0.11095436079645714, pvalue=0.0005848267448603055), men: SpearmanrResult(correlation=0.11504369222990082, pvalue=4.28960553784828e-09), sim353: SpearmanrResult(correlation=0.1318387925484482, pvalue=0.01867078025276011), nearest to monster: ['monster', 'blade', 'tail', 'boat', 'nickname', 'owner', 'leg', 'plain', 'pole', 'ghost']

epoch: 0, iter: 64100, loss: 30.403200149536133
epoch: 0, iter: 64200, loss: 31.01869010925293
epoch: 0, iter: 64300, loss: 30.85900115966797
epoch: 0, iter: 64400, loss: 31.06339454650879
epoch: 0, iter: 64500, loss: 31.443498611450195
epoch: 0, iter: 64600, loss: 30.922685623168945
epoch: 0, iter: 64700, loss: 30.92823028564453
epoch: 0, iter: 64800, loss: 30.95685577392578
epoch: 0, iter: 64900, loss: 31.249370574951172
epoch: 0, iter: 65000, loss: 31.283973693847656
epoch: 0, iter: 65100, loss: 31.421056747436523
epoch: 0, iter: 65200, loss: 31.271799087524414
epoch: 0, iter: 65300, loss: 31.055686950683594
epoch: 0, iter: 65400, loss: 31.06484603881836
epoch: 0, iter: 65500, loss: 31.523380279541016
epoch: 0, iter: 65600, loss: 30.86985969543457
epoch: 0, iter: 65700, loss: 31.431381225585938
epoch: 0, iter: 65800, loss: 30.828258514404297
epoch: 0, iter: 65900, loss: 30.777324676513672
epoch: 0, iter: 66000, loss: 30.793434143066406
epoch: 0, iteration: 66000, simlex-999: SpearmanrResult(correlation=0.11336484593643655, pvalue=0.000441848134889879), men: SpearmanrResult(correlation=0.11607771416192764, pvalue=3.108529516895944e-09), sim353: SpearmanrResult(correlation=0.13567689693977825, pvalue=0.015471766603603733), nearest to monster: ['monster', 'blade', 'tail', 'boat', 'plain', 'mouse', 'leg', 'dubbed', 'angel', 'signature']

epoch: 0, iter: 66100, loss: 30.86065101623535
epoch: 0, iter: 66200, loss: 30.876815795898438
epoch: 0, iter: 66300, loss: 31.069660186767578
epoch: 0, iter: 66400, loss: 30.777523040771484
epoch: 0, iter: 66500, loss: 31.19533920288086
epoch: 0, iter: 66600, loss: 30.554855346679688
epoch: 0, iter: 66700, loss: 30.99230194091797
epoch: 0, iter: 66800, loss: 31.07242202758789
epoch: 0, iter: 66900, loss: 30.73615264892578
epoch: 0, iter: 67000, loss: 31.139455795288086
epoch: 0, iter: 67100, loss: 30.713550567626953
epoch: 0, iter: 67200, loss: 31.377769470214844
epoch: 0, iter: 67300, loss: 31.406234741210938
epoch: 0, iter: 67400, loss: 30.695165634155273
epoch: 0, iter: 67500, loss: 31.24422836303711
epoch: 0, iter: 67600, loss: 30.47709083557129
epoch: 0, iter: 67700, loss: 30.54576301574707
epoch: 0, iter: 67800, loss: 31.31440544128418
epoch: 0, iter: 67900, loss: 30.91347312927246
epoch: 0, iter: 68000, loss: 31.376529693603516
epoch: 0, iteration: 68000, simlex-999: SpearmanrResult(correlation=0.11452703856698428, pvalue=0.00038523421818805733), men: SpearmanrResult(correlation=0.11616177986176482, pvalue=3.027823236694649e-09), sim353: SpearmanrResult(correlation=0.13998669447166648, pvalue=0.012461259148780957), nearest to monster: ['monster', 'boat', 'blade', 'signature', 'tail', 'dubbed', 'mouse', 'cave', 'angel', 'replacing']

epoch: 0, iter: 68100, loss: 30.246761322021484
epoch: 0, iter: 68200, loss: 31.031679153442383
epoch: 0, iter: 68300, loss: 31.415462493896484
epoch: 0, iter: 68400, loss: 30.809803009033203
epoch: 0, iter: 68500, loss: 31.034048080444336
epoch: 0, iter: 68600, loss: 31.134660720825195
epoch: 0, iter: 68700, loss: 31.344093322753906
epoch: 0, iter: 68800, loss: 31.488487243652344
epoch: 0, iter: 68900, loss: 31.44832992553711
epoch: 0, iter: 69000, loss: 30.69766616821289
epoch: 0, iter: 69100, loss: 31.20623016357422
epoch: 0, iter: 69200, loss: 31.305984497070312
epoch: 0, iter: 69300, loss: 30.8837947845459
epoch: 0, iter: 69400, loss: 30.787147521972656
epoch: 0, iter: 69500, loss: 30.73443603515625
epoch: 0, iter: 69600, loss: 30.5230712890625
epoch: 0, iter: 69700, loss: 30.885122299194336
epoch: 0, iter: 69800, loss: 30.608633041381836
epoch: 0, iter: 69900, loss: 31.044784545898438
epoch: 0, iter: 70000, loss: 30.79353141784668
epoch: 0, iteration: 70000, simlex-999: SpearmanrResult(correlation=0.11447412364241306, pvalue=0.00038765739483096796), men: SpearmanrResult(correlation=0.11756886224004129, pvalue=1.944056651206125e-09), sim353: SpearmanrResult(correlation=0.1422861985318877, pvalue=0.011076632617477473), nearest to monster: ['monster', 'boat', 'mouse', 'blade', 'angel', 'replacing', 'leg', 'signature', 'legendary', 'tail']

epoch: 0, iter: 70100, loss: 31.068965911865234
epoch: 0, iter: 70200, loss: 31.09067153930664
epoch: 0, iter: 70300, loss: 30.815410614013672
epoch: 0, iter: 70400, loss: 31.200820922851562
epoch: 0, iter: 70500, loss: 30.970481872558594
epoch: 0, iter: 70600, loss: 30.677066802978516
epoch: 0, iter: 70700, loss: 31.553955078125
epoch: 0, iter: 70800, loss: 30.71514892578125
epoch: 0, iter: 70900, loss: 30.628828048706055
epoch: 0, iter: 71000, loss: 30.579029083251953
epoch: 0, iter: 71100, loss: 30.71631622314453
epoch: 0, iter: 71200, loss: 31.383621215820312
epoch: 0, iter: 71300, loss: 30.815113067626953
epoch: 0, iter: 71400, loss: 31.219520568847656
epoch: 0, iter: 71500, loss: 30.879241943359375
epoch: 0, iter: 71600, loss: 30.864967346191406
epoch: 0, iter: 71700, loss: 31.130123138427734
epoch: 0, iter: 71800, loss: 31.275012969970703
epoch: 0, iter: 71900, loss: 30.728538513183594
epoch: 0, iter: 72000, loss: 30.295608520507812
epoch: 0, iteration: 72000, simlex-999: SpearmanrResult(correlation=0.11509252437598057, pvalue=0.00036020627011201973), men: SpearmanrResult(correlation=0.11721625827550092, pvalue=2.173406746828479e-09), sim353: SpearmanrResult(correlation=0.15003741548009542, pvalue=0.007357771752434236), nearest to monster: ['monster', 'boat', 'leg', 'angel', 'replacing', 'pole', 'legendary', 'tail', 'mouse', 'signature']

epoch: 0, iter: 72100, loss: 30.66713523864746
epoch: 0, iter: 72200, loss: 30.61351776123047
epoch: 0, iter: 72300, loss: 31.320636749267578
epoch: 0, iter: 72400, loss: 31.034809112548828
epoch: 0, iter: 72500, loss: 31.062036514282227
epoch: 0, iter: 72600, loss: 30.442829132080078
epoch: 0, iter: 72700, loss: 30.91510581970215
epoch: 0, iter: 72800, loss: 30.70620346069336
epoch: 0, iter: 72900, loss: 30.421703338623047
epoch: 0, iter: 73000, loss: 30.53826141357422
epoch: 0, iter: 73100, loss: 30.770679473876953
epoch: 0, iter: 73200, loss: 31.04900360107422
epoch: 0, iter: 73300, loss: 30.795854568481445
epoch: 0, iter: 73400, loss: 31.299104690551758
epoch: 0, iter: 73500, loss: 30.484947204589844
epoch: 0, iter: 73600, loss: 30.79161834716797
epoch: 0, iter: 73700, loss: 30.636621475219727
epoch: 0, iter: 73800, loss: 31.00129508972168
epoch: 0, iter: 73900, loss: 30.91973114013672
epoch: 0, iter: 74000, loss: 31.55290985107422
epoch: 0, iteration: 74000, simlex-999: SpearmanrResult(correlation=0.11672803148915531, pvalue=0.0002961005658581428), men: SpearmanrResult(correlation=0.11817601695076835, pvalue=1.6031687449902205e-09), sim353: SpearmanrResult(correlation=0.15298232562148392, pvalue=0.006267834790300931), nearest to monster: ['monster', 'angel', 'legendary', 'leg', 'boat', 'replacing', 'tail', 'dubbed', 'cave', 'pole']

epoch: 0, iter: 74100, loss: 31.169639587402344
epoch: 0, iter: 74200, loss: 30.829368591308594
epoch: 0, iter: 74300, loss: 30.788063049316406
epoch: 0, iter: 74400, loss: 30.632108688354492
epoch: 0, iter: 74500, loss: 30.72389030456543
epoch: 0, iter: 74600, loss: 30.648719787597656
epoch: 0, iter: 74700, loss: 31.583736419677734
epoch: 0, iter: 74800, loss: 30.765384674072266
epoch: 0, iter: 74900, loss: 30.931472778320312
epoch: 0, iter: 75000, loss: 30.993127822875977
epoch: 0, iter: 75100, loss: 30.643539428710938
epoch: 0, iter: 75200, loss: 30.458595275878906
epoch: 0, iter: 75300, loss: 30.298744201660156
epoch: 0, iter: 75400, loss: 30.824840545654297
epoch: 0, iter: 75500, loss: 31.22673797607422
epoch: 0, iter: 75600, loss: 30.644241333007812
epoch: 0, iter: 75700, loss: 30.66327667236328
epoch: 0, iter: 75800, loss: 31.115676879882812
epoch: 0, iter: 75900, loss: 30.466846466064453
epoch: 0, iter: 76000, loss: 30.9396915435791
epoch: 0, iteration: 76000, simlex-999: SpearmanrResult(correlation=0.11596381543200739, pvalue=0.00032459762253928627), men: SpearmanrResult(correlation=0.11879493045996324, pvalue=1.3158101883470934e-09), sim353: SpearmanrResult(correlation=0.15550872005311292, pvalue=0.005450575993741813), nearest to monster: ['monster', 'angel', 'blade', 'replacing', 'boat', 'legendary', 'dubbed', 'leg', 'pole', 'tail']

epoch: 0, iter: 76100, loss: 30.884302139282227
epoch: 0, iter: 76200, loss: 30.992034912109375
epoch: 0, iter: 76300, loss: 30.93535041809082
epoch: 0, iter: 76400, loss: 31.227296829223633
epoch: 0, iter: 76500, loss: 30.600688934326172
epoch: 0, iter: 76600, loss: 30.734973907470703
epoch: 0, iter: 76700, loss: 31.285720825195312
epoch: 0, iter: 76800, loss: 30.783761978149414
epoch: 0, iter: 76900, loss: 31.069557189941406
epoch: 0, iter: 77000, loss: 31.12335205078125
epoch: 0, iter: 77100, loss: 30.547470092773438
epoch: 0, iter: 77200, loss: 30.63747215270996
epoch: 0, iter: 77300, loss: 30.923892974853516
epoch: 0, iter: 77400, loss: 30.970041275024414
epoch: 0, iter: 77500, loss: 31.035385131835938
epoch: 0, iter: 77600, loss: 30.704097747802734
epoch: 0, iter: 77700, loss: 30.910247802734375
epoch: 0, iter: 77800, loss: 30.7044734954834
epoch: 0, iter: 77900, loss: 30.660982131958008
epoch: 0, iter: 78000, loss: 30.560914993286133
epoch: 0, iteration: 78000, simlex-999: SpearmanrResult(correlation=0.11666061112723143, pvalue=0.00029851725007368435), men: SpearmanrResult(correlation=0.12038941573817902, pvalue=7.873392979563775e-10), sim353: SpearmanrResult(correlation=0.15946717336677685, pvalue=0.004361303455838544), nearest to monster: ['monster', 'angel', 'legendary', 'blade', 'boat', 'leg', 'replacing', 'signature', 'tail', 'epic']

epoch: 0, iter: 78100, loss: 31.013874053955078
epoch: 0, iter: 78200, loss: 30.866724014282227
epoch: 0, iter: 78300, loss: 30.68968391418457
epoch: 0, iter: 78400, loss: 31.183582305908203
epoch: 0, iter: 78500, loss: 31.27776336669922
epoch: 0, iter: 78600, loss: 30.777645111083984
epoch: 0, iter: 78700, loss: 31.012584686279297
epoch: 0, iter: 78800, loss: 31.096542358398438
epoch: 0, iter: 78900, loss: 31.362171173095703
epoch: 0, iter: 79000, loss: 30.738359451293945
epoch: 0, iter: 79100, loss: 31.230934143066406
epoch: 0, iter: 79200, loss: 31.539630889892578
epoch: 0, iter: 79300, loss: 30.64007568359375
epoch: 0, iter: 79400, loss: 30.633243560791016
epoch: 0, iter: 79500, loss: 30.83672332763672
epoch: 0, iter: 79600, loss: 30.83807945251465
epoch: 0, iter: 79700, loss: 30.863502502441406
epoch: 0, iter: 79800, loss: 31.011892318725586
epoch: 0, iter: 79900, loss: 30.918609619140625
epoch: 0, iter: 80000, loss: 30.889875411987305
epoch: 0, iteration: 80000, simlex-999: SpearmanrResult(correlation=0.11674166145672565, pvalue=0.0002956142280039947), men: SpearmanrResult(correlation=0.12076227513300175, pvalue=6.975679989657089e-10), sim353: SpearmanrResult(correlation=0.16263542871751985, pvalue=0.00363550206573609), nearest to monster: ['monster', 'blade', 'boat', 'leg', 'legendary', 'bird', 'angel', 'tail', 'signature', 'replacing']

epoch: 0, iter: 80100, loss: 30.875946044921875
epoch: 0, iter: 80200, loss: 31.290252685546875
epoch: 0, iter: 80300, loss: 30.575260162353516
epoch: 0, iter: 80400, loss: 31.11186981201172
epoch: 0, iter: 80500, loss: 30.141450881958008
epoch: 0, iter: 80600, loss: 30.628923416137695
epoch: 0, iter: 80700, loss: 29.730871200561523
epoch: 0, iter: 80800, loss: 30.972171783447266
epoch: 0, iter: 80900, loss: 30.97983169555664
epoch: 0, iter: 81000, loss: 31.120412826538086
epoch: 0, iter: 81100, loss: 31.14563751220703
epoch: 0, iter: 81200, loss: 30.718021392822266
epoch: 0, iter: 81300, loss: 31.257009506225586
epoch: 0, iter: 81400, loss: 30.679397583007812
epoch: 0, iter: 81500, loss: 30.84437370300293
epoch: 0, iter: 81600, loss: 31.431678771972656
epoch: 0, iter: 81700, loss: 30.983497619628906
epoch: 0, iter: 81800, loss: 30.411563873291016
epoch: 0, iter: 81900, loss: 30.98554801940918
epoch: 0, iter: 82000, loss: 30.836700439453125
epoch: 0, iteration: 82000, simlex-999: SpearmanrResult(correlation=0.11957387123233242, pvalue=0.0002092803818694985), men: SpearmanrResult(correlation=0.12247627027840459, pvalue=3.979672482221647e-10), sim353: SpearmanrResult(correlation=0.16490918370598598, pvalue=0.0031840146668676477), nearest to monster: ['monster', 'blade', 'bird', 'boat', 'leg', 'legendary', 'angel', 'tail', 'signature', 'brand']

epoch: 0, iter: 82100, loss: 31.439632415771484
epoch: 0, iter: 82200, loss: 31.005916595458984
epoch: 0, iter: 82300, loss: 30.159162521362305
epoch: 0, iter: 82400, loss: 31.28647232055664
epoch: 0, iter: 82500, loss: 31.14252281188965
epoch: 0, iter: 82600, loss: 30.4515323638916
epoch: 0, iter: 82700, loss: 31.12192153930664
epoch: 0, iter: 82800, loss: 31.36789894104004
epoch: 0, iter: 82900, loss: 30.705039978027344
epoch: 0, iter: 83000, loss: 30.586198806762695
epoch: 0, iter: 83100, loss: 30.898677825927734
epoch: 0, iter: 83200, loss: 30.465381622314453
epoch: 0, iter: 83300, loss: 30.524826049804688
epoch: 0, iter: 83400, loss: 30.76988410949707
epoch: 0, iter: 83500, loss: 31.108976364135742
epoch: 0, iter: 83600, loss: 30.428987503051758
epoch: 0, iter: 83700, loss: 31.005210876464844
epoch: 0, iter: 83800, loss: 30.407583236694336
epoch: 0, iter: 83900, loss: 30.31291961669922
epoch: 0, iter: 84000, loss: 30.384244918823242
epoch: 0, iteration: 84000, simlex-999: SpearmanrResult(correlation=0.11985731732845852, pvalue=0.00020208569630055274), men: SpearmanrResult(correlation=0.12383970685303505, pvalue=2.5324734087303694e-10), sim353: SpearmanrResult(correlation=0.16816942668263687, pvalue=0.002625098004568829), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'boat', 'legendary', 'bird', 'tail', 'signature', 'epic']

epoch: 0, iter: 84100, loss: 30.388629913330078
epoch: 0, iter: 84200, loss: 30.61890983581543
epoch: 0, iter: 84300, loss: 30.860755920410156
epoch: 0, iter: 84400, loss: 30.6082763671875
epoch: 0, iter: 84500, loss: 30.51343536376953
epoch: 0, iter: 84600, loss: 31.175479888916016
epoch: 0, iter: 84700, loss: 30.97124481201172
epoch: 0, iter: 84800, loss: 30.84918975830078
epoch: 0, iter: 84900, loss: 30.95672035217285
epoch: 0, iter: 85000, loss: 31.12570571899414
epoch: 0, iter: 85100, loss: 31.057252883911133
epoch: 0, iter: 85200, loss: 30.39339828491211
epoch: 0, iter: 85300, loss: 30.523571014404297
epoch: 0, iter: 85400, loss: 30.765701293945312
epoch: 0, iter: 85500, loss: 30.65972137451172
epoch: 0, iter: 85600, loss: 30.2365779876709
epoch: 0, iter: 85700, loss: 31.060688018798828
epoch: 0, iter: 85800, loss: 31.084121704101562
epoch: 0, iter: 85900, loss: 30.77812957763672
epoch: 0, iter: 86000, loss: 30.55185890197754
epoch: 0, iteration: 86000, simlex-999: SpearmanrResult(correlation=0.12072190676944367, pvalue=0.00018154682975915078), men: SpearmanrResult(correlation=0.1252523395746619, pvalue=1.577244824410371e-10), sim353: SpearmanrResult(correlation=0.1690460146471711, pvalue=0.002490881483585671), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'boat', 'tail', 'bird', 'mirror', 'legendary', 'signature']

epoch: 0, iter: 86100, loss: 30.656890869140625
epoch: 0, iter: 86200, loss: 30.80274200439453
epoch: 0, iter: 86300, loss: 30.992799758911133
epoch: 0, iter: 86400, loss: 30.460365295410156
epoch: 0, iter: 86500, loss: 30.55353546142578
epoch: 0, iter: 86600, loss: 31.388164520263672
epoch: 0, iter: 86700, loss: 30.856948852539062
epoch: 0, iter: 86800, loss: 30.76443099975586
epoch: 0, iter: 86900, loss: 30.570655822753906
epoch: 0, iter: 87000, loss: 30.948423385620117
epoch: 0, iter: 87100, loss: 30.856409072875977
epoch: 0, iter: 87200, loss: 30.930587768554688
epoch: 0, iter: 87300, loss: 30.785308837890625
epoch: 0, iter: 87400, loss: 30.77594757080078
epoch: 0, iter: 87500, loss: 30.602954864501953
epoch: 0, iter: 87600, loss: 31.219999313354492
epoch: 0, iter: 87700, loss: 30.640804290771484
epoch: 0, iter: 87800, loss: 31.12940788269043
epoch: 0, iter: 87900, loss: 30.826904296875
epoch: 0, iter: 88000, loss: 30.990097045898438
epoch: 0, iteration: 88000, simlex-999: SpearmanrResult(correlation=0.12147024702674274, pvalue=0.0001653680990202469), men: SpearmanrResult(correlation=0.12604951290417668, pvalue=1.2045725972710165e-10), sim353: SpearmanrResult(correlation=0.1693454875469321, pvalue=0.002446480354677961), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'bird', 'boat', 'signature', 'legendary', 'mirror', 'owner']

epoch: 0, iter: 88100, loss: 30.87779426574707
epoch: 0, iter: 88200, loss: 30.53211212158203
epoch: 0, iter: 88300, loss: 30.86421012878418
epoch: 0, iter: 88400, loss: 30.66036605834961
epoch: 0, iter: 88500, loss: 30.340608596801758
epoch: 0, iter: 88600, loss: 30.53420639038086
epoch: 0, iter: 88700, loss: 31.032270431518555
epoch: 0, iter: 88800, loss: 30.652175903320312
epoch: 0, iter: 88900, loss: 31.2420654296875
epoch: 0, iter: 89000, loss: 31.169876098632812
epoch: 0, iter: 89100, loss: 30.760807037353516
epoch: 0, iter: 89200, loss: 31.122560501098633
epoch: 0, iter: 89300, loss: 30.895538330078125
epoch: 0, iter: 89400, loss: 30.56373405456543
epoch: 0, iter: 89500, loss: 30.996185302734375
epoch: 0, iter: 89600, loss: 30.380939483642578
epoch: 0, iter: 89700, loss: 31.11984634399414
epoch: 0, iter: 89800, loss: 30.738248825073242
epoch: 0, iter: 89900, loss: 30.822444915771484
epoch: 0, iter: 90000, loss: 31.190614700317383
epoch: 0, iteration: 90000, simlex-999: SpearmanrResult(correlation=0.12146351761533054, pvalue=0.00016550735138900002), men: SpearmanrResult(correlation=0.12797080964385293, pvalue=6.246963375652522e-11), sim353: SpearmanrResult(correlation=0.1730852603381537, pvalue=0.0019495550082259915), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'signature', 'bird', 'boat', 'tail', 'legendary', 'mirror']

epoch: 0, iter: 90100, loss: 31.01602554321289
epoch: 0, iter: 90200, loss: 30.99297523498535
epoch: 0, iter: 90300, loss: 31.247032165527344
epoch: 0, iter: 90400, loss: 31.122554779052734
epoch: 0, iter: 90500, loss: 30.871871948242188
epoch: 0, iter: 90600, loss: 30.537988662719727
epoch: 0, iter: 90700, loss: 30.66657066345215
epoch: 0, iter: 90800, loss: 30.967605590820312
epoch: 0, iter: 90900, loss: 30.71727180480957
epoch: 0, iter: 91000, loss: 30.835491180419922
epoch: 0, iter: 91100, loss: 30.330137252807617
epoch: 0, iter: 91200, loss: 30.791658401489258
epoch: 0, iter: 91300, loss: 31.337520599365234
epoch: 0, iter: 91400, loss: 30.702518463134766
epoch: 0, iter: 91500, loss: 30.312820434570312
epoch: 0, iter: 91600, loss: 30.737586975097656
epoch: 0, iter: 91700, loss: 30.993764877319336
epoch: 0, iter: 91800, loss: 30.754323959350586
epoch: 0, iter: 91900, loss: 30.35256004333496
epoch: 0, iter: 92000, loss: 31.06475257873535
epoch: 0, iteration: 92000, simlex-999: SpearmanrResult(correlation=0.12049492050252572, pvalue=0.00018674058599766732), men: SpearmanrResult(correlation=0.1287830873875849, pvalue=4.7187134855481034e-11), sim353: SpearmanrResult(correlation=0.17778969038281117, pvalue=0.0014557762316129456), nearest to monster: ['monster', 'leg', 'blade', 'angel', 'mirror', 'signature', 'tail', 'bird', 'boat', 'legendary']

epoch: 0, iter: 92100, loss: 30.96708869934082
epoch: 0, iter: 92200, loss: 30.94014549255371
epoch: 0, iter: 92300, loss: 30.560161590576172
epoch: 0, iter: 92400, loss: 30.880233764648438
epoch: 0, iter: 92500, loss: 30.613208770751953
epoch: 0, iter: 92600, loss: 30.25086212158203
epoch: 0, iter: 92700, loss: 30.913589477539062
epoch: 0, iter: 92800, loss: 30.727537155151367
epoch: 0, iter: 92900, loss: 30.70943832397461
epoch: 0, iter: 93000, loss: 30.733348846435547
epoch: 0, iter: 93100, loss: 31.102148056030273
epoch: 0, iter: 93200, loss: 31.044496536254883
epoch: 0, iter: 93300, loss: 30.72943115234375
epoch: 0, iter: 93400, loss: 30.99721336364746
epoch: 0, iter: 93500, loss: 30.689409255981445
epoch: 0, iter: 93600, loss: 31.005870819091797
epoch: 0, iter: 93700, loss: 30.852521896362305
epoch: 0, iter: 93800, loss: 31.096954345703125
epoch: 0, iter: 93900, loss: 30.707332611083984
epoch: 0, iter: 94000, loss: 31.069786071777344
epoch: 0, iteration: 94000, simlex-999: SpearmanrResult(correlation=0.12324490593078305, pvalue=0.0001322502794958162), men: SpearmanrResult(correlation=0.13004699883992418, pvalue=3.038858436063571e-11), sim353: SpearmanrResult(correlation=0.17592250950560948, pvalue=0.0016361090865993084), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'mirror', 'tail', 'signature', 'bird', 'boat', 'legendary']

epoch: 0, iter: 94100, loss: 31.337974548339844
epoch: 0, iter: 94200, loss: 30.744091033935547
epoch: 0, iter: 94300, loss: 30.94021987915039
epoch: 0, iter: 94400, loss: 30.518966674804688
epoch: 0, iter: 94500, loss: 30.852733612060547
epoch: 0, iter: 94600, loss: 30.059051513671875
epoch: 0, iter: 94700, loss: 31.083663940429688
epoch: 0, iter: 94800, loss: 30.699934005737305
epoch: 0, iter: 94900, loss: 30.6217041015625
epoch: 0, iter: 95000, loss: 30.698474884033203
epoch: 0, iter: 95100, loss: 30.759702682495117
epoch: 0, iter: 95200, loss: 31.207561492919922
epoch: 0, iter: 95300, loss: 30.635595321655273
epoch: 0, iter: 95400, loss: 30.824134826660156
epoch: 0, iter: 95500, loss: 30.59402847290039
epoch: 0, iter: 95600, loss: 29.989118576049805
epoch: 0, iter: 95700, loss: 31.089012145996094
epoch: 0, iter: 95800, loss: 30.574230194091797
epoch: 0, iter: 95900, loss: 31.234333038330078
epoch: 0, iter: 96000, loss: 30.883787155151367
epoch: 0, iteration: 96000, simlex-999: SpearmanrResult(correlation=0.12461901598392905, pvalue=0.00011100775081614734), men: SpearmanrResult(correlation=0.13040298596867406, pvalue=2.6825511352924495e-11), sim353: SpearmanrResult(correlation=0.17773632636453598, pvalue=0.0014606662591213036), nearest to monster: ['monster', 'blade', 'angel', 'leg', 'mirror', 'bird', 'boat', 'tail', 'signature', 'harp']

epoch: 0, iter: 96100, loss: 30.677898406982422
epoch: 0, iter: 96200, loss: 30.666500091552734
epoch: 0, iter: 96300, loss: 30.890090942382812
epoch: 0, iter: 96400, loss: 30.862751007080078
epoch: 0, iter: 96500, loss: 30.74241828918457
epoch: 0, iter: 96600, loss: 31.067392349243164
epoch: 0, iter: 96700, loss: 30.09185791015625
epoch: 0, iter: 96800, loss: 30.830251693725586
epoch: 0, iter: 96900, loss: 30.507057189941406
epoch: 0, iter: 97000, loss: 30.755821228027344
epoch: 0, iter: 97100, loss: 30.22985076904297
epoch: 0, iter: 97200, loss: 30.947574615478516
epoch: 0, iter: 97300, loss: 30.583507537841797
epoch: 0, iter: 97400, loss: 30.67584991455078
epoch: 0, iter: 97500, loss: 31.08060073852539
epoch: 0, iter: 97600, loss: 30.564102172851562
epoch: 0, iter: 97700, loss: 30.59963607788086
epoch: 0, iter: 97800, loss: 31.315624237060547
epoch: 0, iter: 97900, loss: 31.017738342285156
epoch: 0, iter: 98000, loss: 30.729049682617188
epoch: 0, iteration: 98000, simlex-999: SpearmanrResult(correlation=0.1246043454563031, pvalue=0.00011121651888022881), men: SpearmanrResult(correlation=0.13216585436099, pvalue=1.4393399261301587e-11), sim353: SpearmanrResult(correlation=0.17839479356905732, pvalue=0.001401368292639592), nearest to monster: ['monster', 'blade', 'angel', 'leg', 'mirror', 'shield', 'bird', 'tail', 'boat', 'signature']

epoch: 0, iter: 98100, loss: 31.072677612304688
epoch: 0, iter: 98200, loss: 30.446178436279297
epoch: 0, iter: 98300, loss: 30.861177444458008
epoch: 0, iter: 98400, loss: 31.360382080078125
epoch: 0, iter: 98500, loss: 31.02007293701172
epoch: 0, iter: 98600, loss: 30.365102767944336
epoch: 0, iter: 98700, loss: 30.617948532104492
epoch: 0, iter: 98800, loss: 30.435665130615234
epoch: 0, iter: 98900, loss: 30.99168586730957
epoch: 0, iter: 99000, loss: 30.720821380615234
epoch: 0, iter: 99100, loss: 30.526050567626953
epoch: 0, iter: 99200, loss: 30.532978057861328
epoch: 0, iter: 99300, loss: 30.427440643310547
epoch: 0, iter: 99400, loss: 30.835657119750977
epoch: 0, iter: 99500, loss: 30.73276138305664
epoch: 0, iter: 99600, loss: 30.321819305419922
epoch: 0, iter: 99700, loss: 30.551624298095703
epoch: 0, iter: 99800, loss: 30.387161254882812
epoch: 0, iter: 99900, loss: 30.957223892211914
epoch: 0, iter: 100000, loss: 30.369234085083008
epoch: 0, iteration: 100000, simlex-999: SpearmanrResult(correlation=0.12513772832585443, pvalue=0.00010385909688140838), men: SpearmanrResult(correlation=0.13298891292774756, pvalue=1.0732164702661443e-11), sim353: SpearmanrResult(correlation=0.18029779924174447, pvalue=0.0012421912502034132), nearest to monster: ['monster', 'blade', 'angel', 'leg', 'bird', 'boat', 'mirror', 'wheel', 'shield', 'tail']

epoch: 0, iter: 100100, loss: 30.41366958618164
epoch: 0, iter: 100200, loss: 30.84021759033203
epoch: 0, iter: 100300, loss: 30.72504234313965
epoch: 0, iter: 100400, loss: 30.86883544921875
epoch: 0, iter: 100500, loss: 30.47562026977539
epoch: 0, iter: 100600, loss: 30.675384521484375
epoch: 0, iter: 100700, loss: 30.502666473388672
epoch: 0, iter: 100800, loss: 30.754810333251953
epoch: 0, iter: 100900, loss: 30.797780990600586
epoch: 0, iter: 101000, loss: 30.811765670776367
epoch: 0, iter: 101100, loss: 30.670616149902344
epoch: 0, iter: 101200, loss: 30.43963050842285
epoch: 0, iter: 101300, loss: 30.744415283203125
epoch: 0, iter: 101400, loss: 30.76150894165039
epoch: 0, iter: 101500, loss: 30.75295066833496
epoch: 0, iter: 101600, loss: 30.873519897460938
epoch: 0, iter: 101700, loss: 30.82119369506836
epoch: 0, iter: 101800, loss: 30.61273193359375
epoch: 0, iter: 101900, loss: 30.886390686035156
epoch: 0, iter: 102000, loss: 30.50117301940918
epoch: 0, iteration: 102000, simlex-999: SpearmanrResult(correlation=0.12658345688637404, pvalue=8.615546339741256e-05), men: SpearmanrResult(correlation=0.13482600193860372, pvalue=5.5374783924142395e-12), sim353: SpearmanrResult(correlation=0.18156136963251196, pvalue=0.0011458677738041929), nearest to monster: ['monster', 'blade', 'angel', 'leg', 'boat', 'shield', 'mirror', 'bird', 'wheel', 'harp']

epoch: 0, iter: 102100, loss: 31.035795211791992
epoch: 0, iter: 102200, loss: 30.630266189575195
epoch: 0, iter: 102300, loss: 30.40416717529297
epoch: 0, iter: 102400, loss: 30.68378448486328
epoch: 0, iter: 102500, loss: 30.96162986755371
epoch: 0, iter: 102600, loss: 30.965835571289062
epoch: 0, iter: 102700, loss: 30.67343521118164
epoch: 0, iter: 102800, loss: 30.438232421875
epoch: 0, iter: 102900, loss: 30.58956527709961
epoch: 0, iter: 103000, loss: 30.720691680908203
epoch: 0, iter: 103100, loss: 31.140766143798828
epoch: 0, iter: 103200, loss: 30.85077667236328
epoch: 0, iter: 103300, loss: 30.857940673828125
epoch: 0, iter: 103400, loss: 30.87981414794922
epoch: 0, iter: 103500, loss: 30.996490478515625
epoch: 0, iter: 103600, loss: 30.80972671508789
epoch: 0, iter: 103700, loss: 30.426483154296875
epoch: 0, iter: 103800, loss: 30.21402931213379
epoch: 0, iter: 103900, loss: 30.624500274658203
epoch: 0, iter: 104000, loss: 30.79989242553711
epoch: 0, iteration: 104000, simlex-999: SpearmanrResult(correlation=0.12685808215829938, pvalue=8.313188130474721e-05), men: SpearmanrResult(correlation=0.13579942549332547, pvalue=3.885480680746222e-12), sim353: SpearmanrResult(correlation=0.18599692712355217, pvalue=0.0008595570664566224), nearest to monster: ['monster', 'blade', 'angel', 'leg', 'camera', 'shield', 'boat', 'mirror', 'harp', 'elephant']

epoch: 0, iter: 104100, loss: 30.856924057006836
epoch: 0, iter: 104200, loss: 30.86968994140625
epoch: 0, iter: 104300, loss: 31.017091751098633
epoch: 0, iter: 104400, loss: 31.06365394592285
epoch: 0, iter: 104500, loss: 31.071155548095703
epoch: 0, iter: 104600, loss: 31.20572280883789
epoch: 0, iter: 104700, loss: 30.488605499267578
epoch: 0, iter: 104800, loss: 30.928695678710938
epoch: 0, iter: 104900, loss: 30.158926010131836
epoch: 0, iter: 105000, loss: 30.583736419677734
epoch: 0, iter: 105100, loss: 30.513572692871094
epoch: 0, iter: 105200, loss: 30.47650718688965
epoch: 0, iter: 105300, loss: 30.839195251464844
epoch: 0, iter: 105400, loss: 30.879192352294922
epoch: 0, iter: 105500, loss: 30.75198745727539
epoch: 0, iter: 105600, loss: 30.641521453857422
epoch: 0, iter: 105700, loss: 30.412517547607422
epoch: 0, iter: 105800, loss: 30.99948501586914
epoch: 0, iter: 105900, loss: 30.492664337158203
epoch: 0, iter: 106000, loss: 30.717350006103516
epoch: 0, iteration: 106000, simlex-999: SpearmanrResult(correlation=0.12920315538833363, pvalue=6.109467641248834e-05), men: SpearmanrResult(correlation=0.13611945131945707, pvalue=3.4563464313462454e-12), sim353: SpearmanrResult(correlation=0.18889686968310743, pvalue=0.0007097623503415265), nearest to monster: ['monster', 'blade', 'leg', 'angel', 'shield', 'camera', 'mirror', 'elephant', 'tube', 'boat']

epoch: 0, iter: 106100, loss: 31.023670196533203
epoch: 0, iter: 106200, loss: 30.297914505004883
epoch: 0, iter: 106300, loss: 30.34014320373535
epoch: 0, iter: 106400, loss: 30.92735481262207
epoch: 0, iter: 106500, loss: 31.24557113647461
epoch: 0, iter: 106600, loss: 30.648012161254883
epoch: 0, iter: 106700, loss: 30.329431533813477
epoch: 0, iter: 106800, loss: 30.593647003173828
epoch: 0, iter: 106900, loss: 30.407344818115234
epoch: 0, iter: 107000, loss: 30.82100486755371
epoch: 0, iter: 107100, loss: 30.74463653564453
epoch: 0, iter: 107200, loss: 30.87047576904297
epoch: 0, iter: 107300, loss: 30.84787940979004
epoch: 0, iter: 107400, loss: 30.763612747192383
epoch: 0, iter: 107500, loss: 30.8703670501709
epoch: 0, iter: 107600, loss: 30.84850311279297
epoch: 0, iter: 107700, loss: 30.902305603027344
epoch: 0, iter: 107800, loss: 30.58749771118164
epoch: 0, iter: 107900, loss: 30.673856735229492
epoch: 0, iter: 108000, loss: 30.83418846130371
epoch: 0, iteration: 108000, simlex-999: SpearmanrResult(correlation=0.13051674310394015, pvalue=5.129496444866806e-05), men: SpearmanrResult(correlation=0.1375467936587797, pvalue=2.0439266128547674e-12), sim353: SpearmanrResult(correlation=0.19379982880274665, pvalue=0.0005102019337649919), nearest to monster: ['monster', 'blade', 'leg', 'camera', 'shield', 'angel', 'elephant', 'mirror', 'ghost', 'boat']

epoch: 0, iter: 108100, loss: 31.052444458007812
epoch: 0, iter: 108200, loss: 30.354251861572266
epoch: 0, iter: 108300, loss: 31.330671310424805
epoch: 0, iter: 108400, loss: 30.84612274169922
epoch: 0, iter: 108500, loss: 30.641740798950195
epoch: 0, iter: 108600, loss: 30.20119857788086
epoch: 0, iter: 108700, loss: 30.793170928955078
epoch: 0, iter: 108800, loss: 30.220489501953125
epoch: 0, iter: 108900, loss: 30.999284744262695
epoch: 0, iter: 109000, loss: 31.053329467773438
epoch: 0, iter: 109100, loss: 30.955081939697266
epoch: 0, iter: 109200, loss: 30.715665817260742
epoch: 0, iter: 109300, loss: 30.646869659423828
epoch: 0, iter: 109400, loss: 30.617048263549805
epoch: 0, iter: 109500, loss: 31.204490661621094
epoch: 0, iter: 109600, loss: 30.811479568481445
epoch: 0, iter: 109700, loss: 30.87088394165039
epoch: 0, iter: 109800, loss: 30.969287872314453
epoch: 0, iter: 109900, loss: 30.64400291442871
epoch: 0, iter: 110000, loss: 30.75538444519043
epoch: 0, iteration: 110000, simlex-999: SpearmanrResult(correlation=0.13088839031890218, pvalue=4.880473123942339e-05), men: SpearmanrResult(correlation=0.13896681910256206, pvalue=1.2053636316763994e-12), sim353: SpearmanrResult(correlation=0.20021881116883977, pvalue=0.0003271445558931211), nearest to monster: ['monster', 'blade', 'camera', 'leg', 'shield', 'elephant', 'tube', 'mirror', 'angel', 'harp']

epoch: 0, iter: 110100, loss: 30.84221076965332
epoch: 0, iter: 110200, loss: 30.212867736816406
epoch: 0, iter: 110300, loss: 30.807170867919922
epoch: 0, iter: 110400, loss: 30.793251037597656
epoch: 0, iter: 110500, loss: 30.331619262695312
epoch: 0, iter: 110600, loss: 31.10693359375
epoch: 0, iter: 110700, loss: 30.852031707763672
epoch: 0, iter: 110800, loss: 30.546293258666992
epoch: 0, iter: 110900, loss: 30.63283920288086
epoch: 0, iter: 111000, loss: 30.74249839782715
epoch: 0, iter: 111100, loss: 30.533628463745117
epoch: 0, iter: 111200, loss: 30.584836959838867
epoch: 0, iter: 111300, loss: 31.051198959350586
epoch: 0, iter: 111400, loss: 30.349002838134766
epoch: 0, iter: 111500, loss: 31.08504295349121
epoch: 0, iter: 111600, loss: 30.422914505004883
epoch: 0, iter: 111700, loss: 30.63071632385254
epoch: 0, iter: 111800, loss: 30.566604614257812
epoch: 0, iter: 111900, loss: 30.24704360961914
epoch: 0, iter: 112000, loss: 30.796680450439453
epoch: 0, iteration: 112000, simlex-999: SpearmanrResult(correlation=0.13249184544404854, pvalue=3.931449417827171e-05), men: SpearmanrResult(correlation=0.13926854062627636, pvalue=1.0766712070412453e-12), sim353: SpearmanrResult(correlation=0.20095993662544465, pvalue=0.00031050339150380995), nearest to monster: ['monster', 'blade', 'camera', 'tube', 'leg', 'shield', 'boat', 'belt', 'elephant', 'mirror']

epoch: 0, iter: 112100, loss: 30.549213409423828
epoch: 0, iter: 112200, loss: 30.482145309448242
epoch: 0, iter: 112300, loss: 30.55345344543457
epoch: 0, iter: 112400, loss: 30.55514907836914
epoch: 0, iter: 112500, loss: 31.361774444580078
epoch: 0, iter: 112600, loss: 30.88861083984375
epoch: 0, iter: 112700, loss: 30.742958068847656
epoch: 0, iter: 112800, loss: 30.31718635559082
epoch: 0, iter: 112900, loss: 30.65258026123047
epoch: 0, iter: 113000, loss: 30.942604064941406
epoch: 0, iter: 113100, loss: 30.787303924560547
epoch: 0, iter: 113200, loss: 30.986019134521484
epoch: 0, iter: 113300, loss: 30.715797424316406
epoch: 0, iter: 113400, loss: 31.07750701904297
epoch: 0, iter: 113500, loss: 30.899442672729492
epoch: 0, iter: 113600, loss: 30.954410552978516
epoch: 0, iter: 113700, loss: 30.243022918701172
epoch: 0, iter: 113800, loss: 30.855615615844727
epoch: 0, iter: 113900, loss: 31.091819763183594
epoch: 0, iter: 114000, loss: 30.61470603942871
epoch: 0, iteration: 114000, simlex-999: SpearmanrResult(correlation=0.1336421747742616, pvalue=3.3613887168809165e-05), men: SpearmanrResult(correlation=0.14026092296561493, pvalue=7.413973866366574e-13), sim353: SpearmanrResult(correlation=0.1998161553945816, pvalue=0.00033653037818287643), nearest to monster: ['monster', 'blade', 'camera', 'tube', 'shield', 'belt', 'harp', 'leg', 'robot', 'elephant']

epoch: 0, iter: 114100, loss: 31.07042121887207
epoch: 0, iter: 114200, loss: 30.676921844482422
epoch: 0, iter: 114300, loss: 30.714401245117188
epoch: 0, iter: 114400, loss: 30.74327850341797
epoch: 0, iter: 114500, loss: 30.866933822631836
epoch: 0, iter: 114600, loss: 30.79370880126953
epoch: 0, iter: 114700, loss: 30.732254028320312
epoch: 0, iter: 114800, loss: 30.914697647094727
epoch: 0, iter: 114900, loss: 29.94063949584961
epoch: 0, iter: 115000, loss: 31.382705688476562
epoch: 0, iter: 115100, loss: 30.673114776611328
epoch: 0, iter: 115200, loss: 30.157482147216797
epoch: 0, iter: 115300, loss: 30.431161880493164
epoch: 0, iter: 115400, loss: 30.79012680053711
epoch: 0, iter: 115500, loss: 30.708446502685547
epoch: 0, iter: 115600, loss: 30.697505950927734
epoch: 0, iter: 115700, loss: 30.43924331665039
epoch: 0, iter: 115800, loss: 30.20172119140625
epoch: 0, iter: 115900, loss: 30.414777755737305
epoch: 0, iter: 116000, loss: 30.588607788085938
epoch: 0, iteration: 116000, simlex-999: SpearmanrResult(correlation=0.13466175544946068, pvalue=2.922495994353612e-05), men: SpearmanrResult(correlation=0.1408288689686598, pvalue=5.981353363565132e-13), sim353: SpearmanrResult(correlation=0.20141259784340124, pvalue=0.0003007309889417829), nearest to monster: ['monster', 'blade', 'camera', 'tube', 'harp', 'shield', 'belt', 'wheel', 'elephant', 'robot']

epoch: 0, iter: 116100, loss: 31.112529754638672
epoch: 0, iter: 116200, loss: 30.93574333190918
epoch: 0, iter: 116300, loss: 30.862598419189453
epoch: 0, iter: 116400, loss: 30.959362030029297
epoch: 0, iter: 116500, loss: 30.90961456298828
epoch: 0, iter: 116600, loss: 30.95020294189453
epoch: 0, iter: 116700, loss: 30.564823150634766
epoch: 0, iter: 116800, loss: 31.208728790283203
epoch: 0, iter: 116900, loss: 30.43271255493164
epoch: 0, iter: 117000, loss: 30.27477264404297
epoch: 0, iter: 117100, loss: 31.123092651367188
epoch: 0, iter: 117200, loss: 30.779741287231445
epoch: 0, iter: 117300, loss: 30.509449005126953
epoch: 0, iter: 117400, loss: 30.905853271484375
epoch: 0, iter: 117500, loss: 30.46228790283203
epoch: 0, iter: 117600, loss: 30.66716766357422
epoch: 0, iter: 117700, loss: 30.286834716796875
epoch: 0, iter: 117800, loss: 30.221879959106445
epoch: 0, iter: 117900, loss: 30.67200469970703
epoch: 0, iter: 118000, loss: 30.90604019165039
epoch: 0, iteration: 118000, simlex-999: SpearmanrResult(correlation=0.13486929926002347, pvalue=2.8400848845646733e-05), men: SpearmanrResult(correlation=0.14109607175728955, pvalue=5.405006472637779e-13), sim353: SpearmanrResult(correlation=0.20408826225621957, pvalue=0.00024858062563288326), nearest to monster: ['monster', 'blade', 'camera', 'tube', 'harp', 'module', 'robot', 'elephant', 'mirror', 'wheel']

epoch: 0, iter: 118100, loss: 31.184240341186523
epoch: 0, iter: 118200, loss: 29.67246437072754
epoch: 0, iter: 118300, loss: 30.052947998046875
epoch: 0, iter: 118400, loss: 30.136735916137695
epoch: 0, iter: 118500, loss: 31.110488891601562
epoch: 0, iter: 118600, loss: 30.923221588134766
epoch: 0, iter: 118700, loss: 30.17416763305664
epoch: 0, iter: 118800, loss: 30.757675170898438
epoch: 0, iter: 118900, loss: 30.749191284179688
epoch: 0, iter: 119000, loss: 30.639251708984375
epoch: 0, iter: 119100, loss: 30.50901985168457
epoch: 0, iter: 119200, loss: 30.917224884033203
epoch: 0, iter: 119300, loss: 30.909788131713867
epoch: 0, iter: 119400, loss: 30.55084228515625
epoch: 0, iter: 119500, loss: 30.522607803344727
epoch: 1, iter: 0, loss: 30.452777862548828
epoch: 1, iteration: 0, simlex-999: SpearmanrResult(correlation=0.13584630300275116, pvalue=2.480900559909309e-05), men: SpearmanrResult(correlation=0.14149671571670952, pvalue=4.641524444417543e-13), sim353: SpearmanrResult(correlation=0.20420525260397665, pvalue=0.00024650552323513203), nearest to monster: ['monster', 'blade', 'camera', 'robot', 'leg', 'module', 'harp', 'mirror', 'boat', 'elephant']

epoch: 1, iter: 100, loss: 30.471439361572266
epoch: 1, iter: 200, loss: 30.330595016479492
epoch: 1, iter: 300, loss: 30.57529067993164
epoch: 1, iter: 400, loss: 30.718156814575195
epoch: 1, iter: 500, loss: 30.709121704101562
epoch: 1, iter: 600, loss: 30.22405242919922
epoch: 1, iter: 700, loss: 31.00029945373535
epoch: 1, iter: 800, loss: 30.500652313232422
epoch: 1, iter: 900, loss: 30.64475440979004
epoch: 1, iter: 1000, loss: 30.245718002319336
epoch: 1, iter: 1100, loss: 30.46042251586914
epoch: 1, iter: 1200, loss: 30.88376235961914
epoch: 1, iter: 1300, loss: 30.545751571655273
epoch: 1, iter: 1400, loss: 30.541282653808594
epoch: 1, iter: 1500, loss: 30.788883209228516
epoch: 1, iter: 1600, loss: 30.412235260009766
epoch: 1, iter: 1700, loss: 30.570415496826172
epoch: 1, iter: 1800, loss: 30.742263793945312
epoch: 1, iter: 1900, loss: 30.20556640625
epoch: 1, iter: 2000, loss: 30.579498291015625
epoch: 1, iteration: 2000, simlex-999: SpearmanrResult(correlation=0.13750886561871162, pvalue=1.9667970854520583e-05), men: SpearmanrResult(correlation=0.14216903853907206, pvalue=3.5913225784003253e-13), sim353: SpearmanrResult(correlation=0.20737145549247832, pvalue=0.00019612168069552233), nearest to monster: ['monster', 'blade', 'camera', 'module', 'robot', 'boat', 'leg', 'elephant', 'harp', 'pen']

epoch: 1, iter: 2100, loss: 31.068511962890625
epoch: 1, iter: 2200, loss: 30.329666137695312
epoch: 1, iter: 2300, loss: 30.718788146972656
epoch: 1, iter: 2400, loss: 30.20919418334961
epoch: 1, iter: 2500, loss: 30.841068267822266
epoch: 1, iter: 2600, loss: 30.234155654907227
epoch: 1, iter: 2700, loss: 30.538684844970703
epoch: 1, iter: 2800, loss: 30.410411834716797
epoch: 1, iter: 2900, loss: 30.57469940185547
epoch: 1, iter: 3000, loss: 30.982160568237305
epoch: 1, iter: 3100, loss: 30.552490234375
epoch: 1, iter: 3200, loss: 30.447053909301758
epoch: 1, iter: 3300, loss: 30.97784996032715
epoch: 1, iter: 3400, loss: 30.28424072265625
epoch: 1, iter: 3500, loss: 30.430091857910156
epoch: 1, iter: 3600, loss: 30.772613525390625
epoch: 1, iter: 3700, loss: 30.817935943603516
epoch: 1, iter: 3800, loss: 31.377342224121094
epoch: 1, iter: 3900, loss: 30.153400421142578
epoch: 1, iter: 4000, loss: 30.621929168701172
epoch: 1, iteration: 4000, simlex-999: SpearmanrResult(correlation=0.13830312006033485, pvalue=1.7586105251314907e-05), men: SpearmanrResult(correlation=0.14305092230879102, pvalue=2.5604571360787646e-13), sim353: SpearmanrResult(correlation=0.21378372070941798, pvalue=0.0001221329627483309), nearest to monster: ['monster', 'blade', 'camera', 'leg', 'robot', 'boat', 'mirror', 'module', 'harp', 'elephant']

epoch: 1, iter: 4100, loss: 30.806547164916992
epoch: 1, iter: 4200, loss: 30.223846435546875
epoch: 1, iter: 4300, loss: 30.79652214050293
epoch: 1, iter: 4400, loss: 30.826208114624023
epoch: 1, iter: 4500, loss: 30.39246940612793
epoch: 1, iter: 4600, loss: 30.316673278808594
epoch: 1, iter: 4700, loss: 30.22970962524414
epoch: 1, iter: 4800, loss: 30.382505416870117
epoch: 1, iter: 4900, loss: 30.906177520751953
epoch: 1, iter: 5000, loss: 30.31850814819336
epoch: 1, iter: 5100, loss: 30.42485809326172
epoch: 1, iter: 5200, loss: 30.402118682861328
epoch: 1, iter: 5300, loss: 30.640960693359375
epoch: 1, iter: 5400, loss: 30.709318161010742
epoch: 1, iter: 5500, loss: 30.756460189819336
epoch: 1, iter: 5600, loss: 30.85149574279785
epoch: 1, iter: 5700, loss: 30.148801803588867
epoch: 1, iter: 5800, loss: 30.126773834228516
epoch: 1, iter: 5900, loss: 29.931812286376953
epoch: 1, iter: 6000, loss: 30.726451873779297
epoch: 1, iteration: 6000, simlex-999: SpearmanrResult(correlation=0.13881144733907377, pvalue=1.6365610755549817e-05), men: SpearmanrResult(correlation=0.14301940225728388, pvalue=2.5917019394881986e-13), sim353: SpearmanrResult(correlation=0.21994801117010684, pvalue=7.642841580302375e-05), nearest to monster: ['monster', 'blade', 'camera', 'robot', 'harp', 'bird', 'boat', 'leg', 'module', 'elephant']

epoch: 1, iter: 6100, loss: 30.620433807373047
epoch: 1, iter: 6200, loss: 30.196815490722656
epoch: 1, iter: 6300, loss: 30.643386840820312
epoch: 1, iter: 6400, loss: 30.523494720458984
epoch: 1, iter: 6500, loss: 30.796640396118164
epoch: 1, iter: 6600, loss: 31.071800231933594
epoch: 1, iter: 6700, loss: 30.77219009399414
epoch: 1, iter: 6800, loss: 30.722896575927734
epoch: 1, iter: 6900, loss: 30.279769897460938
epoch: 1, iter: 7000, loss: 30.445314407348633
epoch: 1, iter: 7100, loss: 30.531850814819336
epoch: 1, iter: 7200, loss: 30.295429229736328
epoch: 1, iter: 7300, loss: 30.24776840209961
epoch: 1, iter: 7400, loss: 30.302000045776367
epoch: 1, iter: 7500, loss: 30.31418800354004
epoch: 1, iter: 7600, loss: 30.70012664794922
epoch: 1, iter: 7700, loss: 30.701053619384766
epoch: 1, iter: 7800, loss: 30.10363006591797
epoch: 1, iter: 7900, loss: 30.491683959960938
epoch: 1, iter: 8000, loss: 30.433917999267578
epoch: 1, iteration: 8000, simlex-999: SpearmanrResult(correlation=0.13980285510214574, pvalue=1.4213297877880229e-05), men: SpearmanrResult(correlation=0.143487017095893, pvalue=2.1643054711694094e-13), sim353: SpearmanrResult(correlation=0.22035570480623015, pvalue=7.40606141206957e-05), nearest to monster: ['monster', 'blade', 'robot', 'camera', 'bird', 'elephant', 'module', 'mine', 'harp', 'boat']

epoch: 1, iter: 8100, loss: 30.638187408447266
epoch: 1, iter: 8200, loss: 30.01629638671875
epoch: 1, iter: 8300, loss: 30.81504249572754
epoch: 1, iter: 8400, loss: 30.378347396850586
epoch: 1, iter: 8500, loss: 30.630828857421875
epoch: 1, iter: 8600, loss: 30.271350860595703
epoch: 1, iter: 8700, loss: 30.721881866455078
epoch: 1, iter: 8800, loss: 30.455726623535156
epoch: 1, iter: 8900, loss: 31.070642471313477
epoch: 1, iter: 9000, loss: 30.86322593688965
epoch: 1, iter: 9100, loss: 30.664705276489258
epoch: 1, iter: 9200, loss: 30.42993927001953
epoch: 1, iter: 9300, loss: 31.07258415222168
epoch: 1, iter: 9400, loss: 30.924095153808594
epoch: 1, iter: 9500, loss: 30.60555076599121
epoch: 1, iter: 9600, loss: 30.54239845275879
epoch: 1, iter: 9700, loss: 30.296348571777344
epoch: 1, iter: 9800, loss: 30.616439819335938
epoch: 1, iter: 9900, loss: 30.72933578491211
epoch: 1, iter: 10000, loss: 30.84246253967285
epoch: 1, iteration: 10000, simlex-999: SpearmanrResult(correlation=0.13918559761336866, pvalue=1.5519239995073766e-05), men: SpearmanrResult(correlation=0.14349097227167568, pvalue=2.1610034817411894e-13), sim353: SpearmanrResult(correlation=0.22154967806826809, pvalue=6.751724609450489e-05), nearest to monster: ['monster', 'blade', 'bird', 'camera', 'robot', 'elephant', 'mine', 'module', 'leg', 'harp']

epoch: 1, iter: 10100, loss: 30.592771530151367
epoch: 1, iter: 10200, loss: 30.892162322998047
epoch: 1, iter: 10300, loss: 31.060081481933594
epoch: 1, iter: 10400, loss: 30.355026245117188
epoch: 1, iter: 10500, loss: 30.683738708496094
epoch: 1, iter: 10600, loss: 30.789752960205078
epoch: 1, iter: 10700, loss: 30.287927627563477
epoch: 1, iter: 10800, loss: 30.678464889526367
epoch: 1, iter: 10900, loss: 30.326526641845703
epoch: 1, iter: 11000, loss: 30.644237518310547
epoch: 1, iter: 11100, loss: 30.537033081054688
epoch: 1, iter: 11200, loss: 30.29986572265625
epoch: 1, iter: 11300, loss: 30.58269500732422
epoch: 1, iter: 11400, loss: 30.60637664794922
epoch: 1, iter: 11500, loss: 30.54550552368164
epoch: 1, iter: 11600, loss: 30.504114151000977
epoch: 1, iter: 11700, loss: 30.25569725036621
epoch: 1, iter: 11800, loss: 30.455440521240234
epoch: 1, iter: 11900, loss: 30.485092163085938
epoch: 1, iter: 12000, loss: 30.729618072509766
epoch: 1, iteration: 12000, simlex-999: SpearmanrResult(correlation=0.14006879986730683, pvalue=1.3683503449346063e-05), men: SpearmanrResult(correlation=0.14369428471253842, pvalue=1.9977679774445803e-13), sim353: SpearmanrResult(correlation=0.22291381910886188, pvalue=6.0708601908239216e-05), nearest to monster: ['monster', 'blade', 'bird', 'robot', 'camera', 'boat', 'harp', 'mine', 'module', 'elephant']

epoch: 1, iter: 12100, loss: 31.003379821777344
epoch: 1, iter: 12200, loss: 30.895633697509766
epoch: 1, iter: 12300, loss: 30.73479461669922
epoch: 1, iter: 12400, loss: 30.40711212158203
epoch: 1, iter: 12500, loss: 30.21118927001953
epoch: 1, iter: 12600, loss: 30.281837463378906
epoch: 1, iter: 12700, loss: 30.417930603027344
epoch: 1, iter: 12800, loss: 29.946765899658203
epoch: 1, iter: 12900, loss: 30.75798797607422
epoch: 1, iter: 13000, loss: 31.17790985107422
epoch: 1, iter: 13100, loss: 30.745189666748047
epoch: 1, iter: 13200, loss: 30.8111572265625
epoch: 1, iter: 13300, loss: 30.84844207763672
epoch: 1, iter: 13400, loss: 30.430782318115234
epoch: 1, iter: 13500, loss: 30.365447998046875
epoch: 1, iter: 13600, loss: 30.273536682128906
epoch: 1, iter: 13700, loss: 30.858108520507812
epoch: 1, iter: 13800, loss: 30.77298927307129
epoch: 1, iter: 13900, loss: 31.031143188476562
epoch: 1, iter: 14000, loss: 30.615827560424805
epoch: 1, iteration: 14000, simlex-999: SpearmanrResult(correlation=0.1403276597185061, pvalue=1.3185922971315998e-05), men: SpearmanrResult(correlation=0.14529215462232734, pvalue=1.0734198813575383e-13), sim353: SpearmanrResult(correlation=0.22418410664878283, pvalue=5.495482632416603e-05), nearest to monster: ['monster', 'blade', 'bird', 'boat', 'robot', 'mine', 'module', 'camera', 'giant', 'harp']

epoch: 1, iter: 14100, loss: 30.48816680908203
epoch: 1, iter: 14200, loss: 30.806354522705078
epoch: 1, iter: 14300, loss: 29.96129035949707
epoch: 1, iter: 14400, loss: 30.932781219482422
epoch: 1, iter: 14500, loss: 30.7196102142334
epoch: 1, iter: 14600, loss: 30.22078514099121
epoch: 1, iter: 14700, loss: 31.128929138183594
epoch: 1, iter: 14800, loss: 29.683853149414062
epoch: 1, iter: 14900, loss: 30.984813690185547
epoch: 1, iter: 15000, loss: 30.461803436279297
epoch: 1, iter: 15100, loss: 30.73785400390625
epoch: 1, iter: 15200, loss: 30.712100982666016
epoch: 1, iter: 15300, loss: 30.466516494750977
epoch: 1, iter: 15400, loss: 30.314067840576172
epoch: 1, iter: 15500, loss: 30.55801773071289
epoch: 1, iter: 15600, loss: 30.428577423095703
epoch: 1, iter: 15700, loss: 31.00663185119629
epoch: 1, iter: 15800, loss: 30.276063919067383
epoch: 1, iter: 15900, loss: 30.50764274597168
epoch: 1, iter: 16000, loss: 30.646820068359375
epoch: 1, iteration: 16000, simlex-999: SpearmanrResult(correlation=0.13965840845801117, pvalue=1.4509180896993527e-05), men: SpearmanrResult(correlation=0.14605203149208384, pvalue=7.969295632331052e-14), sim353: SpearmanrResult(correlation=0.22762533947762578, pvalue=4.1841877221568024e-05), nearest to monster: ['monster', 'blade', 'robot', 'bird', 'boat', 'camera', 'mine', 'harp', 'pen', 'module']

epoch: 1, iter: 16100, loss: 30.518444061279297
epoch: 1, iter: 16200, loss: 30.669857025146484
epoch: 1, iter: 16300, loss: 30.55596351623535
epoch: 1, iter: 16400, loss: 30.038494110107422
epoch: 1, iter: 16500, loss: 30.565519332885742
epoch: 1, iter: 16600, loss: 30.724658966064453
epoch: 1, iter: 16700, loss: 30.56858253479004
epoch: 1, iter: 16800, loss: 30.65789031982422
epoch: 1, iter: 16900, loss: 30.921201705932617
epoch: 1, iter: 17000, loss: 30.809478759765625
epoch: 1, iter: 17100, loss: 30.42205238342285
epoch: 1, iter: 17200, loss: 30.399263381958008
epoch: 1, iter: 17300, loss: 31.051158905029297
epoch: 1, iter: 17400, loss: 30.570343017578125
epoch: 1, iter: 17500, loss: 30.416711807250977
epoch: 1, iter: 17600, loss: 30.80805778503418
epoch: 1, iter: 17700, loss: 30.25259780883789
epoch: 1, iter: 17800, loss: 30.286285400390625
epoch: 1, iter: 17900, loss: 30.53017807006836
epoch: 1, iter: 18000, loss: 30.383750915527344
epoch: 1, iteration: 18000, simlex-999: SpearmanrResult(correlation=0.1394123296964421, pvalue=1.5026797423339371e-05), men: SpearmanrResult(correlation=0.14637243756010673, pvalue=7.025458630976344e-14), sim353: SpearmanrResult(correlation=0.22732157506590558, pvalue=4.2868219856181685e-05), nearest to monster: ['monster', 'blade', 'camera', 'robot', 'bird', 'harp', 'boat', 'mine', 'pen', 'triangle']

epoch: 1, iter: 18100, loss: 30.563156127929688
epoch: 1, iter: 18200, loss: 30.877361297607422
epoch: 1, iter: 18300, loss: 30.779144287109375
epoch: 1, iter: 18400, loss: 30.79215431213379
epoch: 1, iter: 18500, loss: 30.42022705078125
epoch: 1, iter: 18600, loss: 31.03143310546875
epoch: 1, iter: 18700, loss: 31.016183853149414
epoch: 1, iter: 18800, loss: 31.11318016052246
epoch: 1, iter: 18900, loss: 30.872020721435547
epoch: 1, iter: 19000, loss: 30.474287033081055
epoch: 1, iter: 19100, loss: 31.39320945739746
epoch: 1, iter: 19200, loss: 31.03207015991211
epoch: 1, iter: 19300, loss: 30.487464904785156
epoch: 1, iter: 19400, loss: 30.420028686523438
epoch: 1, iter: 19500, loss: 30.75209617614746
epoch: 1, iter: 19600, loss: 29.88393783569336
epoch: 1, iter: 19700, loss: 30.5760440826416
epoch: 1, iter: 19800, loss: 30.56399917602539
epoch: 1, iter: 19900, loss: 30.256702423095703
epoch: 1, iter: 20000, loss: 30.273393630981445
epoch: 1, iteration: 20000, simlex-999: SpearmanrResult(correlation=0.14223311686242013, pvalue=1.0018980772884857e-05), men: SpearmanrResult(correlation=0.14753960651141446, pvalue=4.428169020670323e-14), sim353: SpearmanrResult(correlation=0.22932179281384227, pvalue=3.652359446164985e-05), nearest to monster: ['monster', 'blade', 'bird', 'robot', 'mine', 'harp', 'boat', 'triangle', 'camera', 'giant']

epoch: 1, iter: 20100, loss: 30.554931640625
epoch: 1, iter: 20200, loss: 30.8705997467041
epoch: 1, iter: 20300, loss: 30.918720245361328
epoch: 1, iter: 20400, loss: 30.68167495727539
epoch: 1, iter: 20500, loss: 30.421613693237305
epoch: 1, iter: 20600, loss: 30.65481948852539
epoch: 1, iter: 20700, loss: 30.48784828186035
epoch: 1, iter: 20800, loss: 30.522476196289062
epoch: 1, iter: 20900, loss: 30.651445388793945
epoch: 1, iter: 21000, loss: 30.873241424560547
epoch: 1, iter: 21100, loss: 30.824703216552734
epoch: 1, iter: 21200, loss: 30.80352783203125
epoch: 1, iter: 21300, loss: 31.21027183532715
epoch: 1, iter: 21400, loss: 30.860301971435547
epoch: 1, iter: 21500, loss: 30.478717803955078
epoch: 1, iter: 21600, loss: 30.538753509521484
epoch: 1, iter: 21700, loss: 31.42760467529297
epoch: 1, iter: 21800, loss: 31.005023956298828
epoch: 1, iter: 21900, loss: 30.42320442199707
epoch: 1, iter: 22000, loss: 30.21063232421875
epoch: 1, iteration: 22000, simlex-999: SpearmanrResult(correlation=0.1428147967792888, pvalue=9.206660938754159e-06), men: SpearmanrResult(correlation=0.14807291207730797, pvalue=3.5817711226237477e-14), sim353: SpearmanrResult(correlation=0.22970919127508704, pvalue=3.5402130162864466e-05), nearest to monster: ['monster', 'blade', 'bird', 'mine', 'robot', 'giant', 'boat', 'harp', 'camera', 'pen']

epoch: 1, iter: 22100, loss: 30.796628952026367
epoch: 1, iter: 22200, loss: 30.456802368164062
epoch: 1, iter: 22300, loss: 30.246700286865234
epoch: 1, iter: 22400, loss: 30.443458557128906
epoch: 1, iter: 22500, loss: 30.49130630493164
epoch: 1, iter: 22600, loss: 30.463035583496094
epoch: 1, iter: 22700, loss: 30.521265029907227
epoch: 1, iter: 22800, loss: 30.170909881591797
epoch: 1, iter: 22900, loss: 30.44857406616211
epoch: 1, iter: 23000, loss: 30.58160972595215
epoch: 1, iter: 23100, loss: 30.80916404724121
epoch: 1, iter: 23200, loss: 30.507898330688477
epoch: 1, iter: 23300, loss: 30.540969848632812
epoch: 1, iter: 23400, loss: 30.272123336791992
epoch: 1, iter: 23500, loss: 30.770973205566406
epoch: 1, iter: 23600, loss: 30.44808006286621
epoch: 1, iter: 23700, loss: 30.266027450561523
epoch: 1, iter: 23800, loss: 30.43990707397461
epoch: 1, iter: 23900, loss: 30.382156372070312
epoch: 1, iter: 24000, loss: 30.580440521240234
epoch: 1, iteration: 24000, simlex-999: SpearmanrResult(correlation=0.14287100755966056, pvalue=9.131580692652058e-06), men: SpearmanrResult(correlation=0.14868537859505318, pvalue=2.804668835689092e-14), sim353: SpearmanrResult(correlation=0.2263985268197262, pvalue=4.613476464710824e-05), nearest to monster: ['monster', 'bird', 'blade', 'giant', 'robot', 'mine', 'pen', 'camera', 'triangle', 'boat']

epoch: 1, iter: 24100, loss: 30.35135841369629
epoch: 1, iter: 24200, loss: 30.669477462768555
epoch: 1, iter: 24300, loss: 30.54084587097168
epoch: 1, iter: 24400, loss: 30.29105567932129
epoch: 1, iter: 24500, loss: 30.614038467407227
epoch: 1, iter: 24600, loss: 30.688926696777344
epoch: 1, iter: 24700, loss: 30.522205352783203
epoch: 1, iter: 24800, loss: 30.543987274169922
epoch: 1, iter: 24900, loss: 31.09662628173828
epoch: 1, iter: 25000, loss: 30.970827102661133
epoch: 1, iter: 25100, loss: 30.467174530029297
epoch: 1, iter: 25200, loss: 30.76879119873047
epoch: 1, iter: 25300, loss: 30.6207218170166
epoch: 1, iter: 25400, loss: 30.490726470947266
epoch: 1, iter: 25500, loss: 30.458398818969727
epoch: 1, iter: 25600, loss: 30.703933715820312
epoch: 1, iter: 25700, loss: 30.121395111083984
epoch: 1, iter: 25800, loss: 30.44470977783203
epoch: 1, iter: 25900, loss: 30.887786865234375
epoch: 1, iter: 26000, loss: 30.558914184570312
epoch: 1, iteration: 26000, simlex-999: SpearmanrResult(correlation=0.1440751174505626, pvalue=7.656767087120004e-06), men: SpearmanrResult(correlation=0.1491477742745481, pvalue=2.3302203512655484e-14), sim353: SpearmanrResult(correlation=0.23077736171791446, pvalue=3.247610265381441e-05), nearest to monster: ['monster', 'blade', 'bird', 'robot', 'mine', 'elephant', 'harp', 'triangle', 'pen', 'reed']

epoch: 1, iter: 26100, loss: 30.59500503540039
epoch: 1, iter: 26200, loss: 30.334857940673828
epoch: 1, iter: 26300, loss: 30.802188873291016
epoch: 1, iter: 26400, loss: 30.327043533325195
epoch: 1, iter: 26500, loss: 30.643577575683594
epoch: 1, iter: 26600, loss: 30.822498321533203
epoch: 1, iter: 26700, loss: 30.609739303588867
epoch: 1, iter: 26800, loss: 30.348751068115234
epoch: 1, iter: 26900, loss: 30.833683013916016
epoch: 1, iter: 27000, loss: 30.705856323242188
epoch: 1, iter: 27100, loss: 30.677705764770508
epoch: 1, iter: 27200, loss: 30.479990005493164
epoch: 1, iter: 27300, loss: 30.481945037841797
epoch: 1, iter: 27400, loss: 30.73859214782715
epoch: 1, iter: 27500, loss: 30.516708374023438
epoch: 1, iter: 27600, loss: 30.041893005371094
epoch: 1, iter: 27700, loss: 30.019962310791016
epoch: 1, iter: 27800, loss: 30.198917388916016
epoch: 1, iter: 27900, loss: 31.1048583984375
epoch: 1, iter: 28000, loss: 30.839107513427734
epoch: 1, iteration: 28000, simlex-999: SpearmanrResult(correlation=0.14549903097533917, pvalue=6.205612354950464e-06), men: SpearmanrResult(correlation=0.14966965306971067, pvalue=1.8890981268115036e-14), sim353: SpearmanrResult(correlation=0.23346739336642225, pvalue=2.6087214499345555e-05), nearest to monster: ['monster', 'robot', 'blade', 'bird', 'mine', 'reed', 'triangle', 'ghost', 'pen', 'harp']

epoch: 1, iter: 28100, loss: 30.756732940673828
epoch: 1, iter: 28200, loss: 30.536327362060547
epoch: 1, iter: 28300, loss: 31.24537467956543
epoch: 1, iter: 28400, loss: 30.601255416870117
epoch: 1, iter: 28500, loss: 30.263526916503906
epoch: 1, iter: 28600, loss: 30.142976760864258
epoch: 1, iter: 28700, loss: 30.747371673583984
epoch: 1, iter: 28800, loss: 30.630146026611328
epoch: 1, iter: 28900, loss: 30.744644165039062
epoch: 1, iter: 29000, loss: 30.7540283203125
epoch: 1, iter: 29100, loss: 30.279701232910156
epoch: 1, iter: 29200, loss: 30.66191291809082
epoch: 1, iter: 29300, loss: 30.671695709228516
epoch: 1, iter: 29400, loss: 30.434926986694336
epoch: 1, iter: 29500, loss: 30.72023582458496
epoch: 1, iter: 29600, loss: 30.603559494018555
epoch: 1, iter: 29700, loss: 30.372743606567383
epoch: 1, iter: 29800, loss: 30.525760650634766
epoch: 1, iter: 29900, loss: 30.840803146362305
epoch: 1, iter: 30000, loss: 30.364925384521484
epoch: 1, iteration: 30000, simlex-999: SpearmanrResult(correlation=0.14568623594428892, pvalue=6.035624214317192e-06), men: SpearmanrResult(correlation=0.1509883492958195, pvalue=1.1079359739743368e-14), sim353: SpearmanrResult(correlation=0.23492314885797097, pvalue=2.3145855166212694e-05), nearest to monster: ['monster', 'robot', 'blade', 'reed', 'mine', 'rod', 'pen', 'bird', 'enigma', 'horn']

epoch: 1, iter: 30100, loss: 30.345247268676758
epoch: 1, iter: 30200, loss: 30.77888298034668
epoch: 1, iter: 30300, loss: 30.320167541503906
epoch: 1, iter: 30400, loss: 30.234546661376953
epoch: 1, iter: 30500, loss: 30.409408569335938
epoch: 1, iter: 30600, loss: 30.470806121826172
epoch: 1, iter: 30700, loss: 30.835254669189453
epoch: 1, iter: 30800, loss: 30.207406997680664
epoch: 1, iter: 30900, loss: 30.916057586669922
epoch: 1, iter: 31000, loss: 30.66683006286621
epoch: 1, iter: 31100, loss: 30.577659606933594
epoch: 1, iter: 31200, loss: 30.580257415771484
epoch: 1, iter: 31300, loss: 30.935575485229492
epoch: 1, iter: 31400, loss: 30.697229385375977
epoch: 1, iter: 31500, loss: 30.42900848388672
epoch: 1, iter: 31600, loss: 30.660232543945312
epoch: 1, iter: 31700, loss: 30.72662353515625
epoch: 1, iter: 31800, loss: 29.9237060546875
epoch: 1, iter: 31900, loss: 30.48178482055664
epoch: 1, iter: 32000, loss: 30.498600006103516
epoch: 1, iteration: 32000, simlex-999: SpearmanrResult(correlation=0.14636174425669718, pvalue=5.458483599637883e-06), men: SpearmanrResult(correlation=0.15148407706724934, pvalue=9.054505875289788e-15), sim353: SpearmanrResult(correlation=0.23389001400066448, pvalue=2.51987417090668e-05), nearest to monster: ['monster', 'blade', 'robot', 'reed', 'mine', 'bird', 'rod', 'giant', 'enigma', 'pen']

epoch: 1, iter: 32100, loss: 31.0069580078125
epoch: 1, iter: 32200, loss: 30.799236297607422
epoch: 1, iter: 32300, loss: 30.04582405090332
epoch: 1, iter: 32400, loss: 30.311925888061523
epoch: 1, iter: 32500, loss: 30.556798934936523
epoch: 1, iter: 32600, loss: 30.72783660888672
epoch: 1, iter: 32700, loss: 30.761629104614258
epoch: 1, iter: 32800, loss: 30.628524780273438
epoch: 1, iter: 32900, loss: 30.40520477294922
epoch: 1, iter: 33000, loss: 30.739730834960938
epoch: 1, iter: 33100, loss: 30.354564666748047
epoch: 1, iter: 33200, loss: 30.47233009338379
epoch: 1, iter: 33300, loss: 31.133811950683594
epoch: 1, iter: 33400, loss: 30.499738693237305
epoch: 1, iter: 33500, loss: 30.797992706298828
epoch: 1, iter: 33600, loss: 30.39028549194336
epoch: 1, iter: 33700, loss: 30.41649627685547
epoch: 1, iter: 33800, loss: 30.45271873474121
epoch: 1, iter: 33900, loss: 30.6096248626709
epoch: 1, iter: 34000, loss: 30.480247497558594
epoch: 1, iteration: 34000, simlex-999: SpearmanrResult(correlation=0.1473754688161237, pvalue=4.69031176866327e-06), men: SpearmanrResult(correlation=0.1529620524813248, pvalue=4.940947898179605e-15), sim353: SpearmanrResult(correlation=0.2346744277518143, pvalue=2.362507911995377e-05), nearest to monster: ['monster', 'blade', 'robot', 'mine', 'bird', 'giant', 'reed', 'rod', 'clown', 'enigma']

epoch: 1, iter: 34100, loss: 30.54153823852539
epoch: 1, iter: 34200, loss: 30.292634963989258
epoch: 1, iter: 34300, loss: 30.777355194091797
epoch: 1, iter: 34400, loss: 30.559667587280273
epoch: 1, iter: 34500, loss: 29.992252349853516
epoch: 1, iter: 34600, loss: 30.286727905273438
epoch: 1, iter: 34700, loss: 30.328699111938477
epoch: 1, iter: 34800, loss: 30.475990295410156
epoch: 1, iter: 34900, loss: 30.71676254272461
epoch: 1, iter: 35000, loss: 30.722000122070312
epoch: 1, iter: 35100, loss: 30.39300537109375
epoch: 1, iter: 35200, loss: 30.483230590820312
epoch: 1, iter: 35300, loss: 30.614459991455078
epoch: 1, iter: 35400, loss: 29.942462921142578
epoch: 1, iter: 35500, loss: 30.2659854888916
epoch: 1, iter: 35600, loss: 30.37142562866211
epoch: 1, iter: 35700, loss: 29.414955139160156
epoch: 1, iter: 35800, loss: 30.6357479095459
epoch: 1, iter: 35900, loss: 30.29971694946289
epoch: 1, iter: 36000, loss: 30.45860481262207
epoch: 1, iteration: 36000, simlex-999: SpearmanrResult(correlation=0.1479277049908929, pvalue=4.316518848130977e-06), men: SpearmanrResult(correlation=0.1534592647777511, pvalue=4.024677001188328e-15), sim353: SpearmanrResult(correlation=0.23541144828393648, pvalue=2.22316752869189e-05), nearest to monster: ['monster', 'blade', 'giant', 'mine', 'bird', 'reed', 'robot', 'rod', 'clown', 'enigma']

epoch: 1, iter: 36100, loss: 30.45232391357422
epoch: 1, iter: 36200, loss: 30.192720413208008
epoch: 1, iter: 36300, loss: 30.173755645751953
epoch: 1, iter: 36400, loss: 30.194917678833008
epoch: 1, iter: 36500, loss: 30.539600372314453
epoch: 1, iter: 36600, loss: 30.462547302246094
epoch: 1, iter: 36700, loss: 30.38189697265625
epoch: 1, iter: 36800, loss: 30.229658126831055
epoch: 1, iter: 36900, loss: 30.55349349975586
epoch: 1, iter: 37000, loss: 30.451251983642578
epoch: 1, iter: 37100, loss: 30.258930206298828
epoch: 1, iter: 37200, loss: 30.375003814697266
epoch: 1, iter: 37300, loss: 30.661685943603516
epoch: 1, iter: 37400, loss: 30.22052001953125
epoch: 1, iter: 37500, loss: 30.464017868041992
epoch: 1, iter: 37600, loss: 30.531139373779297
epoch: 1, iter: 37700, loss: 30.161556243896484
epoch: 1, iter: 37800, loss: 30.280288696289062
epoch: 1, iter: 37900, loss: 30.579071044921875
epoch: 1, iter: 38000, loss: 30.68809700012207
epoch: 1, iteration: 38000, simlex-999: SpearmanrResult(correlation=0.14842186148466413, pvalue=4.006345881913544e-06), men: SpearmanrResult(correlation=0.1535692327051543, pvalue=3.845826952077038e-15), sim353: SpearmanrResult(correlation=0.24061126807866595, pvalue=1.4397516474117237e-05), nearest to monster: ['monster', 'blade', 'bird', 'mine', 'robot', 'reed', 'giant', 'ghost', 'enigma', 'harp']

epoch: 1, iter: 38100, loss: 30.60862922668457
epoch: 1, iter: 38200, loss: 30.42845916748047
epoch: 1, iter: 38300, loss: 30.334047317504883
epoch: 1, iter: 38400, loss: 30.224014282226562
epoch: 1, iter: 38500, loss: 30.38711166381836
epoch: 1, iter: 38600, loss: 30.579326629638672
epoch: 1, iter: 38700, loss: 30.49921417236328
epoch: 1, iter: 38800, loss: 30.80820083618164
epoch: 1, iter: 38900, loss: 31.00635528564453
epoch: 1, iter: 39000, loss: 30.365596771240234
epoch: 1, iter: 39100, loss: 30.78212547302246
epoch: 1, iter: 39200, loss: 30.845741271972656
epoch: 1, iter: 39300, loss: 30.45212173461914
epoch: 1, iter: 39400, loss: 30.290306091308594
epoch: 1, iter: 39500, loss: 30.514169692993164
epoch: 1, iter: 39600, loss: 30.55576515197754
epoch: 1, iter: 39700, loss: 31.103591918945312
epoch: 1, iter: 39800, loss: 30.821365356445312
epoch: 1, iter: 39900, loss: 30.697998046875
epoch: 1, iter: 40000, loss: 30.70787811279297
epoch: 1, iteration: 40000, simlex-999: SpearmanrResult(correlation=0.14840725941493657, pvalue=4.015197419076501e-06), men: SpearmanrResult(correlation=0.15357136041153727, pvalue=3.842444769779768e-15), sim353: SpearmanrResult(correlation=0.24335858208245514, pvalue=1.1399247512803185e-05), nearest to monster: ['monster', 'blade', 'bird', 'reed', 'robot', 'ghost', 'mine', 'enigma', 'rod', 'giant']

epoch: 1, iter: 40100, loss: 30.93113899230957
epoch: 1, iter: 40200, loss: 30.542682647705078
epoch: 1, iter: 40300, loss: 30.148162841796875
epoch: 1, iter: 40400, loss: 30.786636352539062
epoch: 1, iter: 40500, loss: 30.51471710205078
epoch: 1, iter: 40600, loss: 30.892784118652344
epoch: 1, iter: 40700, loss: 30.750877380371094
epoch: 1, iter: 40800, loss: 30.910011291503906
epoch: 1, iter: 40900, loss: 30.417316436767578
epoch: 1, iter: 41000, loss: 30.877635955810547
epoch: 1, iter: 41100, loss: 31.21525764465332
epoch: 1, iter: 41200, loss: 30.61797332763672
epoch: 1, iter: 41300, loss: 29.999229431152344
epoch: 1, iter: 41400, loss: 30.419879913330078
epoch: 1, iter: 41500, loss: 30.497615814208984
epoch: 1, iter: 41600, loss: 31.086910247802734
epoch: 1, iter: 41700, loss: 31.012123107910156
epoch: 1, iter: 41800, loss: 30.67609977722168
epoch: 1, iter: 41900, loss: 30.800514221191406
epoch: 1, iter: 42000, loss: 30.609895706176758
epoch: 1, iteration: 42000, simlex-999: SpearmanrResult(correlation=0.15010702795733427, pvalue=3.101153292343823e-06), men: SpearmanrResult(correlation=0.15464227684833873, pvalue=2.4637832880461148e-15), sim353: SpearmanrResult(correlation=0.2412430532740497, pvalue=1.3648100607040392e-05), nearest to monster: ['monster', 'blade', 'ghost', 'bird', 'robot', 'mine', 'reed', 'enigma', 'tiger', 'reads']

epoch: 1, iter: 42100, loss: 30.431297302246094
epoch: 1, iter: 42200, loss: 31.107322692871094
epoch: 1, iter: 42300, loss: 30.734027862548828
epoch: 1, iter: 42400, loss: 30.269119262695312
epoch: 1, iter: 42500, loss: 30.10508155822754
epoch: 1, iter: 42600, loss: 30.353633880615234
epoch: 1, iter: 42700, loss: 30.593158721923828
epoch: 1, iter: 42800, loss: 30.670429229736328
epoch: 1, iter: 42900, loss: 30.44912338256836
epoch: 1, iter: 43000, loss: 30.147382736206055
epoch: 1, iter: 43100, loss: 30.15931510925293
epoch: 1, iter: 43200, loss: 30.213638305664062
epoch: 1, iter: 43300, loss: 30.583301544189453
epoch: 1, iter: 43400, loss: 30.891719818115234
epoch: 1, iter: 43500, loss: 30.8968448638916
epoch: 1, iter: 43600, loss: 30.235538482666016
epoch: 1, iter: 43700, loss: 30.293712615966797
epoch: 1, iter: 43800, loss: 30.817930221557617
epoch: 1, iter: 43900, loss: 30.755901336669922
epoch: 1, iter: 44000, loss: 30.5270938873291
epoch: 1, iteration: 44000, simlex-999: SpearmanrResult(correlation=0.15040984462509274, pvalue=2.9607903887451717e-06), men: SpearmanrResult(correlation=0.15462659465737233, pvalue=2.479925723603805e-15), sim353: SpearmanrResult(correlation=0.24184031978628343, pvalue=1.29738022188437e-05), nearest to monster: ['monster', 'ghost', 'robot', 'blade', 'bird', 'reed', 'enigma', 'mine', 'pen', 'giant']

epoch: 1, iter: 44100, loss: 30.67951011657715
epoch: 1, iter: 44200, loss: 30.391963958740234
epoch: 1, iter: 44300, loss: 30.283348083496094
epoch: 1, iter: 44400, loss: 30.486854553222656
epoch: 1, iter: 44500, loss: 29.995058059692383
epoch: 1, iter: 44600, loss: 29.919422149658203
epoch: 1, iter: 44700, loss: 30.681249618530273
epoch: 1, iter: 44800, loss: 30.176651000976562
epoch: 1, iter: 44900, loss: 31.011629104614258
epoch: 1, iter: 45000, loss: 30.286090850830078
epoch: 1, iter: 45100, loss: 30.60489273071289
epoch: 1, iter: 45200, loss: 30.306713104248047
epoch: 1, iter: 45300, loss: 30.408802032470703
epoch: 1, iter: 45400, loss: 30.530902862548828
epoch: 1, iter: 45500, loss: 30.43999481201172
epoch: 1, iter: 45600, loss: 30.87363624572754
epoch: 1, iter: 45700, loss: 30.33321762084961
epoch: 1, iter: 45800, loss: 30.374361038208008
epoch: 1, iter: 45900, loss: 30.656036376953125
epoch: 1, iter: 46000, loss: 30.552873611450195
epoch: 1, iteration: 46000, simlex-999: SpearmanrResult(correlation=0.1506515900016934, pvalue=2.8531245356788605e-06), men: SpearmanrResult(correlation=0.15536691339728612, pvalue=1.820665609880339e-15), sim353: SpearmanrResult(correlation=0.24012203571531798, pvalue=1.5004498206655285e-05), nearest to monster: ['monster', 'robot', 'bird', 'blade', 'mine', 'ghost', 'enigma', 'triangle', 'reed', 'pen']

epoch: 1, iter: 46100, loss: 30.70708656311035
epoch: 1, iter: 46200, loss: 30.461339950561523
epoch: 1, iter: 46300, loss: 30.217098236083984
epoch: 1, iter: 46400, loss: 30.840656280517578
epoch: 1, iter: 46500, loss: 30.493894577026367
epoch: 1, iter: 46600, loss: 30.61757469177246
epoch: 1, iter: 46700, loss: 30.641674041748047
epoch: 1, iter: 46800, loss: 30.795719146728516
epoch: 1, iter: 46900, loss: 30.51910400390625
epoch: 1, iter: 47000, loss: 30.65304946899414
epoch: 1, iter: 47100, loss: 30.11658477783203
epoch: 1, iter: 47200, loss: 30.48131561279297
epoch: 1, iter: 47300, loss: 30.761260986328125
epoch: 1, iter: 47400, loss: 30.574722290039062
epoch: 1, iter: 47500, loss: 30.653053283691406
epoch: 1, iter: 47600, loss: 30.582984924316406
epoch: 1, iter: 47700, loss: 30.558650970458984
epoch: 1, iter: 47800, loss: 30.786725997924805
epoch: 1, iter: 47900, loss: 30.690078735351562
epoch: 1, iter: 48000, loss: 30.940021514892578
epoch: 1, iteration: 48000, simlex-999: SpearmanrResult(correlation=0.1512458867092362, pvalue=2.6041543817031568e-06), men: SpearmanrResult(correlation=0.1557567213740912, pvalue=1.5463267464958923e-15), sim353: SpearmanrResult(correlation=0.24230566895264108, pvalue=1.247048744341614e-05), nearest to monster: ['monster', 'robot', 'mine', 'blade', 'ghost', 'bird', 'triangle', 'pen', 'mirror', 'reed']

epoch: 1, iter: 48100, loss: 30.569496154785156
epoch: 1, iter: 48200, loss: 30.488758087158203
epoch: 1, iter: 48300, loss: 30.724409103393555
epoch: 1, iter: 48400, loss: 30.37924575805664
epoch: 1, iter: 48500, loss: 30.688404083251953
epoch: 1, iter: 48600, loss: 31.19538688659668
epoch: 1, iter: 48700, loss: 31.03116226196289
epoch: 1, iter: 48800, loss: 30.401390075683594
epoch: 1, iter: 48900, loss: 30.71941566467285
epoch: 1, iter: 49000, loss: 30.60601043701172
epoch: 1, iter: 49100, loss: 30.593238830566406
epoch: 1, iter: 49200, loss: 30.418926239013672
epoch: 1, iter: 49300, loss: 30.479490280151367
epoch: 1, iter: 49400, loss: 30.65471649169922
epoch: 1, iter: 49500, loss: 31.25054359436035
epoch: 1, iter: 49600, loss: 30.942890167236328
epoch: 1, iter: 49700, loss: 30.643369674682617
epoch: 1, iter: 49800, loss: 30.203311920166016
epoch: 1, iter: 49900, loss: 30.408056259155273
epoch: 1, iter: 50000, loss: 30.49826431274414
epoch: 1, iteration: 50000, simlex-999: SpearmanrResult(correlation=0.1525143841833279, pvalue=2.140523243449846e-06), men: SpearmanrResult(correlation=0.15705781004352684, pvalue=8.938073731951767e-16), sim353: SpearmanrResult(correlation=0.24303933091018648, pvalue=1.1714453427503608e-05), nearest to monster: ['monster', 'robot', 'mine', 'ghost', 'blade', 'triangle', 'bird', 'mirror', 'pen', 'trilogy']

epoch: 1, iter: 50100, loss: 30.582521438598633
epoch: 1, iter: 50200, loss: 30.166404724121094
epoch: 1, iter: 50300, loss: 30.79269790649414
epoch: 1, iter: 50400, loss: 30.7398738861084
epoch: 1, iter: 50500, loss: 30.5670108795166
epoch: 1, iter: 50600, loss: 30.718910217285156
epoch: 1, iter: 50700, loss: 30.94159507751465
epoch: 1, iter: 50800, loss: 30.046207427978516
epoch: 1, iter: 50900, loss: 30.098331451416016
epoch: 1, iter: 51000, loss: 29.920578002929688
epoch: 1, iter: 51100, loss: 30.722366333007812
epoch: 1, iter: 51200, loss: 30.87830352783203
epoch: 1, iter: 51300, loss: 30.5864200592041
epoch: 1, iter: 51400, loss: 30.589786529541016
epoch: 1, iter: 51500, loss: 30.31692886352539
epoch: 1, iter: 51600, loss: 30.661989212036133
epoch: 1, iter: 51700, loss: 30.14481544494629
epoch: 1, iter: 51800, loss: 30.520448684692383
epoch: 1, iter: 51900, loss: 30.797508239746094
epoch: 1, iter: 52000, loss: 29.767147064208984
epoch: 1, iteration: 52000, simlex-999: SpearmanrResult(correlation=0.15373255141051995, pvalue=1.7705182477987938e-06), men: SpearmanrResult(correlation=0.1585061793559117, pvalue=4.829025134421122e-16), sim353: SpearmanrResult(correlation=0.24509030045172758, pvalue=9.825140052862682e-06), nearest to monster: ['monster', 'robot', 'blade', 'mine', 'ghost', 'triangle', 'pen', 'reed', 'mirror', 'storyline']

epoch: 1, iter: 52100, loss: 30.682071685791016
epoch: 1, iter: 52200, loss: 30.016061782836914
epoch: 1, iter: 52300, loss: 30.534984588623047
epoch: 1, iter: 52400, loss: 30.56295394897461
epoch: 1, iter: 52500, loss: 30.030967712402344
epoch: 1, iter: 52600, loss: 30.3729305267334
epoch: 1, iter: 52700, loss: 30.135780334472656
epoch: 1, iter: 52800, loss: 30.442276000976562
epoch: 1, iter: 52900, loss: 30.262731552124023
epoch: 1, iter: 53000, loss: 30.31039047241211
epoch: 1, iter: 53100, loss: 31.22720718383789
epoch: 1, iter: 53200, loss: 30.35527801513672
epoch: 1, iter: 53300, loss: 30.849201202392578
epoch: 1, iter: 53400, loss: 30.32939910888672
epoch: 1, iter: 53500, loss: 30.342052459716797
epoch: 1, iter: 53600, loss: 30.57972526550293
epoch: 1, iter: 53700, loss: 29.98486328125
epoch: 1, iter: 53800, loss: 30.098655700683594
epoch: 1, iter: 53900, loss: 30.3535213470459
epoch: 1, iter: 54000, loss: 30.042743682861328
epoch: 1, iteration: 54000, simlex-999: SpearmanrResult(correlation=0.1536032686694018, pvalue=1.8066656752367236e-06), men: SpearmanrResult(correlation=0.1585990419061131, pvalue=4.641201609227958e-16), sim353: SpearmanrResult(correlation=0.25060825189386676, pvalue=6.074459332227015e-06), nearest to monster: ['monster', 'robot', 'ghost', 'blade', 'mine', 'triangle', 'bird', 'pen', 'giant', 'trilogy']

epoch: 1, iter: 54100, loss: 30.99134063720703
epoch: 1, iter: 54200, loss: 30.4207820892334
epoch: 1, iter: 54300, loss: 29.99266815185547
epoch: 1, iter: 54400, loss: 30.288175582885742
epoch: 1, iter: 54500, loss: 30.557764053344727
epoch: 1, iter: 54600, loss: 30.272037506103516
epoch: 1, iter: 54700, loss: 30.78939437866211
epoch: 1, iter: 54800, loss: 30.318763732910156
epoch: 1, iter: 54900, loss: 30.315868377685547
epoch: 1, iter: 55000, loss: 30.52359962463379
epoch: 1, iter: 55100, loss: 30.504047393798828
epoch: 1, iter: 55200, loss: 30.359554290771484
epoch: 1, iter: 55300, loss: 30.453826904296875
epoch: 1, iter: 55400, loss: 30.830547332763672
epoch: 1, iter: 55500, loss: 30.533924102783203
epoch: 1, iter: 55600, loss: 30.538036346435547
epoch: 1, iter: 55700, loss: 30.60943603515625
epoch: 1, iter: 55800, loss: 30.83778190612793
epoch: 1, iter: 55900, loss: 30.720783233642578
epoch: 1, iter: 56000, loss: 30.150545120239258
epoch: 1, iteration: 56000, simlex-999: SpearmanrResult(correlation=0.1549713586940779, pvalue=1.457568750556874e-06), men: SpearmanrResult(correlation=0.15844155102847718, pvalue=4.96414018000047e-16), sim353: SpearmanrResult(correlation=0.24769058352404524, pvalue=7.843960247464432e-06), nearest to monster: ['monster', 'robot', 'ghost', 'giant', 'pen', 'mine', 'bird', 'blade', 'storyline', 'triangle']

epoch: 1, iter: 56100, loss: 30.171369552612305
epoch: 1, iter: 56200, loss: 30.735065460205078
epoch: 1, iter: 56300, loss: 30.493385314941406
epoch: 1, iter: 56400, loss: 30.45469856262207
epoch: 1, iter: 56500, loss: 30.479145050048828
epoch: 1, iter: 56600, loss: 30.517457962036133
epoch: 1, iter: 56700, loss: 30.358793258666992
epoch: 1, iter: 56800, loss: 30.938446044921875
epoch: 1, iter: 56900, loss: 30.858768463134766
epoch: 1, iter: 57000, loss: 30.36663818359375
epoch: 1, iter: 57100, loss: 30.539148330688477
epoch: 1, iter: 57200, loss: 30.83847427368164
epoch: 1, iter: 57300, loss: 30.67807388305664
epoch: 1, iter: 57400, loss: 30.247142791748047
epoch: 1, iter: 57500, loss: 30.21233367919922
epoch: 1, iter: 57600, loss: 30.43688201904297
epoch: 1, iter: 57700, loss: 30.402862548828125
epoch: 1, iter: 57800, loss: 30.552614212036133
epoch: 1, iter: 57900, loss: 30.985126495361328
epoch: 1, iter: 58000, loss: 30.84340476989746
epoch: 1, iteration: 58000, simlex-999: SpearmanrResult(correlation=0.1567349094604098, pvalue=1.1021343002031157e-06), men: SpearmanrResult(correlation=0.15902652554699048, pvalue=3.865317045437844e-16), sim353: SpearmanrResult(correlation=0.24948667457270562, pvalue=6.704232943262648e-06), nearest to monster: ['monster', 'robot', 'giant', 'pen', 'ghost', 'mine', 'bird', 'clown', 'reed', 'mirror']

epoch: 1, iter: 58100, loss: 30.185630798339844
epoch: 1, iter: 58200, loss: 30.426095962524414
epoch: 1, iter: 58300, loss: 30.398298263549805
epoch: 1, iter: 58400, loss: 30.785690307617188
epoch: 1, iter: 58500, loss: 30.762544631958008
epoch: 1, iter: 58600, loss: 30.656660079956055
epoch: 1, iter: 58700, loss: 30.267963409423828
epoch: 1, iter: 58800, loss: 30.180572509765625
epoch: 1, iter: 58900, loss: 30.352771759033203
epoch: 1, iter: 59000, loss: 30.799413681030273
epoch: 1, iter: 59100, loss: 30.187427520751953
epoch: 1, iter: 59200, loss: 30.583654403686523
epoch: 1, iter: 59300, loss: 30.478708267211914
epoch: 1, iter: 59400, loss: 30.59113121032715
epoch: 1, iter: 59500, loss: 30.526996612548828
epoch: 1, iter: 59600, loss: 29.880691528320312
epoch: 1, iter: 59700, loss: 30.175827026367188
epoch: 1, iter: 59800, loss: 31.251361846923828
epoch: 1, iter: 59900, loss: 30.632102966308594
epoch: 1, iter: 60000, loss: 30.681888580322266
epoch: 1, iteration: 60000, simlex-999: SpearmanrResult(correlation=0.15766878472714893, pvalue=9.493068391257752e-07), men: SpearmanrResult(correlation=0.1599384328989951, pvalue=2.6120469754271507e-16), sim353: SpearmanrResult(correlation=0.2506047067318135, pvalue=6.0763581426292506e-06), nearest to monster: ['monster', 'robot', 'giant', 'pen', 'ghost', 'cow', 'storyline', 'blade', 'bird', 'mine']

epoch: 1, iter: 60100, loss: 30.331680297851562
epoch: 1, iter: 60200, loss: 30.630077362060547
epoch: 1, iter: 60300, loss: 30.40506362915039
epoch: 1, iter: 60400, loss: 30.622928619384766
epoch: 1, iter: 60500, loss: 30.441761016845703
epoch: 1, iter: 60600, loss: 30.31716537475586
epoch: 1, iter: 60700, loss: 30.88729476928711
epoch: 1, iter: 60800, loss: 30.464248657226562
epoch: 1, iter: 60900, loss: 30.365467071533203
epoch: 1, iter: 61000, loss: 30.315540313720703
epoch: 1, iter: 61100, loss: 30.57969856262207
epoch: 1, iter: 61200, loss: 30.44414520263672
epoch: 1, iter: 61300, loss: 30.557884216308594
epoch: 1, iter: 61400, loss: 30.003623962402344
epoch: 1, iter: 61500, loss: 30.530445098876953
epoch: 1, iter: 61600, loss: 30.768972396850586
epoch: 1, iter: 61700, loss: 30.325469970703125
epoch: 1, iter: 61800, loss: 30.635547637939453
epoch: 1, iter: 61900, loss: 30.08446502685547
epoch: 1, iter: 62000, loss: 30.466468811035156
epoch: 1, iteration: 62000, simlex-999: SpearmanrResult(correlation=0.1572838048897373, pvalue=1.0096640956970655e-06), men: SpearmanrResult(correlation=0.16137007881415613, pvalue=1.405264112416452e-16), sim353: SpearmanrResult(correlation=0.25097676215992787, pvalue=5.8801359002641125e-06), nearest to monster: ['monster', 'robot', 'pen', 'ghost', 'giant', 'cow', 'mine', 'storyline', 'bird', 'blade']

epoch: 1, iter: 62100, loss: 30.18429946899414
epoch: 1, iter: 62200, loss: 30.559833526611328
epoch: 1, iter: 62300, loss: 30.80440902709961
epoch: 1, iter: 62400, loss: 30.450206756591797
epoch: 1, iter: 62500, loss: 30.552818298339844
epoch: 1, iter: 62600, loss: 30.82094383239746
epoch: 1, iter: 62700, loss: 30.254344940185547
epoch: 1, iter: 62800, loss: 30.72846221923828
epoch: 1, iter: 62900, loss: 30.654434204101562
epoch: 1, iter: 63000, loss: 30.073328018188477
epoch: 1, iter: 63100, loss: 30.521087646484375
epoch: 1, iter: 63200, loss: 30.763505935668945
epoch: 1, iter: 63300, loss: 30.240577697753906
epoch: 1, iter: 63400, loss: 30.317813873291016
epoch: 1, iter: 63500, loss: 30.43919563293457
epoch: 1, iter: 63600, loss: 30.885873794555664
epoch: 1, iter: 63700, loss: 30.13918685913086
epoch: 1, iter: 63800, loss: 30.71306610107422
epoch: 1, iter: 63900, loss: 30.3992919921875
epoch: 1, iter: 64000, loss: 30.920867919921875
epoch: 1, iteration: 64000, simlex-999: SpearmanrResult(correlation=0.15698834744605628, pvalue=1.0584701505049052e-06), men: SpearmanrResult(correlation=0.1624835806175362, pvalue=8.642929806246007e-17), sim353: SpearmanrResult(correlation=0.24978148278555448, pvalue=6.532925572818505e-06), nearest to monster: ['monster', 'robot', 'pen', 'storyline', 'giant', 'ghost', 'triangle', 'blade', 'cow', 'clown']

epoch: 1, iter: 64100, loss: 30.346240997314453
epoch: 1, iter: 64200, loss: 30.147878646850586
epoch: 1, iter: 64300, loss: 30.068510055541992
epoch: 1, iter: 64400, loss: 30.74982452392578
epoch: 1, iter: 64500, loss: 30.429706573486328
epoch: 1, iter: 64600, loss: 30.37489128112793
epoch: 1, iter: 64700, loss: 30.758684158325195
epoch: 1, iter: 64800, loss: 30.118595123291016
epoch: 1, iter: 64900, loss: 30.60893440246582
epoch: 1, iter: 65000, loss: 30.800472259521484
epoch: 1, iter: 65100, loss: 30.511920928955078
epoch: 1, iter: 65200, loss: 30.794395446777344
epoch: 1, iter: 65300, loss: 30.49081802368164
epoch: 1, iter: 65400, loss: 30.53462791442871
epoch: 1, iter: 65500, loss: 30.544513702392578
epoch: 1, iter: 65600, loss: 30.508167266845703
epoch: 1, iter: 65700, loss: 30.721145629882812
epoch: 1, iter: 65800, loss: 30.107677459716797
epoch: 1, iter: 65900, loss: 30.400524139404297
epoch: 1, iter: 66000, loss: 30.216068267822266
epoch: 1, iteration: 66000, simlex-999: SpearmanrResult(correlation=0.15907493049754254, pvalue=7.569762376508921e-07), men: SpearmanrResult(correlation=0.1622197008816302, pvalue=9.70111944701947e-17), sim353: SpearmanrResult(correlation=0.25537220334352345, pvalue=3.974470732999653e-06), nearest to monster: ['monster', 'robot', 'giant', 'pen', 'storyline', 'triangle', 'clown', 'reed', 'ghost', 'blade']

epoch: 1, iter: 66100, loss: 30.723974227905273
epoch: 1, iter: 66200, loss: 30.21212387084961
epoch: 1, iter: 66300, loss: 30.621620178222656
epoch: 1, iter: 66400, loss: 31.26927947998047
epoch: 1, iter: 66500, loss: 30.06582260131836
epoch: 1, iter: 66600, loss: 30.753280639648438
epoch: 1, iter: 66700, loss: 30.463829040527344
epoch: 1, iter: 66800, loss: 30.498855590820312
epoch: 1, iter: 66900, loss: 30.842039108276367
epoch: 1, iter: 67000, loss: 29.559223175048828
epoch: 1, iter: 67100, loss: 30.619075775146484
epoch: 1, iter: 67200, loss: 30.44180679321289
epoch: 1, iter: 67300, loss: 30.190532684326172
epoch: 1, iter: 67400, loss: 30.0849552154541
epoch: 1, iter: 67500, loss: 30.109207153320312
epoch: 1, iter: 67600, loss: 30.49976348876953
epoch: 1, iter: 67700, loss: 30.740123748779297
epoch: 1, iter: 67800, loss: 30.926212310791016
epoch: 1, iter: 67900, loss: 30.441604614257812
epoch: 1, iter: 68000, loss: 30.73117446899414
epoch: 1, iteration: 68000, simlex-999: SpearmanrResult(correlation=0.15894696899386346, pvalue=7.727968829631284e-07), men: SpearmanrResult(correlation=0.16225855404638603, pvalue=9.537652911995123e-17), sim353: SpearmanrResult(correlation=0.2561217252365703, pvalue=3.715012465818261e-06), nearest to monster: ['monster', 'robot', 'giant', 'clown', 'pen', 'triangle', 'killer', 'storyline', 'reed', 'vampire']

epoch: 1, iter: 68100, loss: 30.470741271972656
epoch: 1, iter: 68200, loss: 30.515743255615234
epoch: 1, iter: 68300, loss: 30.583833694458008
epoch: 1, iter: 68400, loss: 29.629898071289062
epoch: 1, iter: 68500, loss: 30.373931884765625
epoch: 1, iter: 68600, loss: 30.927085876464844
epoch: 1, iter: 68700, loss: 30.736282348632812
epoch: 1, iter: 68800, loss: 30.456012725830078
epoch: 1, iter: 68900, loss: 30.077430725097656
epoch: 1, iter: 69000, loss: 30.592777252197266
epoch: 1, iter: 69100, loss: 30.24420738220215
epoch: 1, iter: 69200, loss: 29.89266586303711
epoch: 1, iter: 69300, loss: 30.14252471923828
epoch: 1, iter: 69400, loss: 30.470155715942383
epoch: 1, iter: 69500, loss: 30.696094512939453
epoch: 1, iter: 69600, loss: 30.263404846191406
epoch: 1, iter: 69700, loss: 30.48387336730957
epoch: 1, iter: 69800, loss: 31.011812210083008
epoch: 1, iter: 69900, loss: 30.354785919189453
epoch: 1, iter: 70000, loss: 30.257991790771484
epoch: 1, iteration: 70000, simlex-999: SpearmanrResult(correlation=0.1599804231196073, pvalue=6.536001505651106e-07), men: SpearmanrResult(correlation=0.16235736400138848, pvalue=9.134061487743441e-17), sim353: SpearmanrResult(correlation=0.2579999147748847, pvalue=3.1339385169295858e-06), nearest to monster: ['monster', 'robot', 'clown', 'giant', 'triangle', 'pen', 'storyline', 'vampire', 'ghost', 'killer']

epoch: 1, iter: 70100, loss: 30.840045928955078
epoch: 1, iter: 70200, loss: 30.389122009277344
epoch: 1, iter: 70300, loss: 31.05836296081543
epoch: 1, iter: 70400, loss: 30.655227661132812
epoch: 1, iter: 70500, loss: 30.400094985961914
epoch: 1, iter: 70600, loss: 30.757015228271484
epoch: 1, iter: 70700, loss: 30.292152404785156
epoch: 1, iter: 70800, loss: 30.084205627441406
epoch: 1, iter: 70900, loss: 30.660961151123047
epoch: 1, iter: 71000, loss: 30.49699592590332
epoch: 1, iter: 71100, loss: 30.42240333557129
epoch: 1, iter: 71200, loss: 29.931034088134766
epoch: 1, iter: 71300, loss: 30.057233810424805
epoch: 1, iter: 71400, loss: 30.548381805419922
epoch: 1, iter: 71500, loss: 30.618526458740234
epoch: 1, iter: 71600, loss: 30.42486000061035
epoch: 1, iter: 71700, loss: 30.71780014038086
epoch: 1, iter: 71800, loss: 30.28960418701172
epoch: 1, iter: 71900, loss: 30.402591705322266
epoch: 1, iter: 72000, loss: 30.8114013671875
epoch: 1, iteration: 72000, simlex-999: SpearmanrResult(correlation=0.16101441806274847, pvalue=5.521495930342118e-07), men: SpearmanrResult(correlation=0.1633857027791665, pvalue=5.814925831866898e-17), sim353: SpearmanrResult(correlation=0.25737857847818424, pvalue=3.3158227988787696e-06), nearest to monster: ['monster', 'robot', 'clown', 'giant', 'triangle', 'killer', 'storyline', 'horn', 'bird', 'reed']

epoch: 1, iter: 72100, loss: 30.883159637451172
epoch: 1, iter: 72200, loss: 30.85186767578125
epoch: 1, iter: 72300, loss: 30.58966636657715
epoch: 1, iter: 72400, loss: 30.387969970703125
epoch: 1, iter: 72500, loss: 29.79990005493164
epoch: 1, iter: 72600, loss: 30.107318878173828
epoch: 1, iter: 72700, loss: 30.190916061401367
epoch: 1, iter: 72800, loss: 30.59153175354004
epoch: 1, iter: 72900, loss: 30.534543991088867
epoch: 1, iter: 73000, loss: 30.79871368408203
epoch: 1, iter: 73100, loss: 30.439369201660156
epoch: 1, iter: 73200, loss: 29.90771484375
epoch: 1, iter: 73300, loss: 29.77695083618164
epoch: 1, iter: 73400, loss: 30.999645233154297
epoch: 1, iter: 73500, loss: 30.230430603027344
epoch: 1, iter: 73600, loss: 30.236263275146484
epoch: 1, iter: 73700, loss: 29.803932189941406
epoch: 1, iter: 73800, loss: 30.766613006591797
epoch: 1, iter: 73900, loss: 30.48684310913086
epoch: 1, iter: 74000, loss: 30.37664222717285
epoch: 1, iteration: 74000, simlex-999: SpearmanrResult(correlation=0.1617221261012147, pvalue=4.916430156015046e-07), men: SpearmanrResult(correlation=0.16269528376899162, pvalue=7.876945371094953e-17), sim353: SpearmanrResult(correlation=0.2568775911038176, pvalue=3.4697706327940896e-06), nearest to monster: ['monster', 'giant', 'clown', 'robot', 'triangle', 'killer', 'horn', 'storyline', 'bird', 'pen']

epoch: 1, iter: 74100, loss: 30.254886627197266
epoch: 1, iter: 74200, loss: 29.888710021972656
epoch: 1, iter: 74300, loss: 30.417236328125
epoch: 1, iter: 74400, loss: 30.457595825195312
epoch: 1, iter: 74500, loss: 31.00020980834961
epoch: 1, iter: 74600, loss: 30.30846405029297
epoch: 1, iter: 74700, loss: 30.387718200683594
epoch: 1, iter: 74800, loss: 30.376087188720703
epoch: 1, iter: 74900, loss: 30.061664581298828
epoch: 1, iter: 75000, loss: 30.370288848876953
epoch: 1, iter: 75100, loss: 30.63956642150879
epoch: 1, iter: 75200, loss: 30.442768096923828
epoch: 1, iter: 75300, loss: 30.677764892578125
epoch: 1, iter: 75400, loss: 30.916812896728516
epoch: 1, iter: 75500, loss: 30.47184944152832
epoch: 1, iter: 75600, loss: 30.52167510986328
epoch: 1, iter: 75700, loss: 30.56326675415039
epoch: 1, iter: 75800, loss: 29.92554473876953
epoch: 1, iter: 75900, loss: 30.573081970214844
epoch: 1, iter: 76000, loss: 30.38399314880371
epoch: 1, iteration: 76000, simlex-999: SpearmanrResult(correlation=0.16169187742526095, pvalue=4.940931234078511e-07), men: SpearmanrResult(correlation=0.1633295111707588, pvalue=5.960650109011135e-17), sim353: SpearmanrResult(correlation=0.25485181087160225, pvalue=4.16468278962377e-06), nearest to monster: ['monster', 'triangle', 'robot', 'clown', 'giant', 'storyline', 'killer', 'pen', 'cow', 'shield']

epoch: 1, iter: 76100, loss: 30.317106246948242
epoch: 1, iter: 76200, loss: 30.010608673095703
epoch: 1, iter: 76300, loss: 29.795989990234375
epoch: 1, iter: 76400, loss: 29.63566780090332
epoch: 1, iter: 76500, loss: 30.61663055419922
epoch: 1, iter: 76600, loss: 30.317100524902344
epoch: 1, iter: 76700, loss: 30.912708282470703
epoch: 1, iter: 76800, loss: 30.493019104003906
epoch: 1, iter: 76900, loss: 30.66164779663086
epoch: 1, iter: 77000, loss: 30.15361213684082
epoch: 1, iter: 77100, loss: 30.159067153930664
epoch: 1, iter: 77200, loss: 30.553518295288086
epoch: 1, iter: 77300, loss: 30.48749542236328
epoch: 1, iter: 77400, loss: 30.554759979248047
epoch: 1, iter: 77500, loss: 30.53312110900879
epoch: 1, iter: 77600, loss: 30.761314392089844
epoch: 1, iter: 77700, loss: 31.00925064086914
epoch: 1, iter: 77800, loss: 30.383312225341797
epoch: 1, iter: 77900, loss: 30.3116455078125
epoch: 1, iter: 78000, loss: 30.420455932617188
epoch: 1, iteration: 78000, simlex-999: SpearmanrResult(correlation=0.16207894522857358, pvalue=4.6360979168089965e-07), men: SpearmanrResult(correlation=0.16372154472938574, pvalue=5.0143933981199564e-17), sim353: SpearmanrResult(correlation=0.2544614698707922, pvalue=4.313024622770899e-06), nearest to monster: ['monster', 'triangle', 'robot', 'clown', 'giant', 'killer', 'pen', 'cow', 'rod', 'bird']

epoch: 1, iter: 78100, loss: 30.672090530395508
epoch: 1, iter: 78200, loss: 30.56073760986328
epoch: 1, iter: 78300, loss: 30.491344451904297
epoch: 1, iter: 78400, loss: 31.016584396362305
epoch: 1, iter: 78500, loss: 30.333904266357422
epoch: 1, iter: 78600, loss: 30.13150405883789
epoch: 1, iter: 78700, loss: 30.275697708129883
epoch: 1, iter: 78800, loss: 29.988712310791016
epoch: 1, iter: 78900, loss: 30.521224975585938
epoch: 1, iter: 79000, loss: 30.032791137695312
epoch: 1, iter: 79100, loss: 30.854816436767578
epoch: 1, iter: 79200, loss: 30.634782791137695
epoch: 1, iter: 79300, loss: 30.44858741760254
epoch: 1, iter: 79400, loss: 30.231046676635742
epoch: 1, iter: 79500, loss: 30.371126174926758
epoch: 1, iter: 79600, loss: 29.989742279052734
epoch: 1, iter: 79700, loss: 30.68444061279297
epoch: 1, iter: 79800, loss: 30.262882232666016
epoch: 1, iter: 79900, loss: 30.341434478759766
epoch: 1, iter: 80000, loss: 29.98185920715332
epoch: 1, iteration: 80000, simlex-999: SpearmanrResult(correlation=0.16252507166083122, pvalue=4.3072065008154985e-07), men: SpearmanrResult(correlation=0.1643962621991115, pvalue=3.720276950472103e-17), sim353: SpearmanrResult(correlation=0.25403922241150295, pvalue=4.479160666059968e-06), nearest to monster: ['monster', 'triangle', 'robot', 'clown', 'giant', 'killer', 'rod', 'pen', 'bull', 'bird']

epoch: 1, iter: 80100, loss: 30.37212371826172
epoch: 1, iter: 80200, loss: 30.76333999633789
epoch: 1, iter: 80300, loss: 30.685781478881836
epoch: 1, iter: 80400, loss: 30.539093017578125
epoch: 1, iter: 80500, loss: 30.19442367553711
epoch: 1, iter: 80600, loss: 31.10211944580078
epoch: 1, iter: 80700, loss: 30.563905715942383
epoch: 1, iter: 80800, loss: 30.892704010009766
epoch: 1, iter: 80900, loss: 30.355934143066406
epoch: 1, iter: 81000, loss: 30.444995880126953
epoch: 1, iter: 81100, loss: 29.747629165649414
epoch: 1, iter: 81200, loss: 30.422718048095703
epoch: 1, iter: 81300, loss: 30.607858657836914
epoch: 1, iter: 81400, loss: 30.45087432861328
epoch: 1, iter: 81500, loss: 30.621387481689453
epoch: 1, iter: 81600, loss: 30.56511878967285
epoch: 1, iter: 81700, loss: 30.036951065063477
epoch: 1, iter: 81800, loss: 29.739484786987305
epoch: 1, iter: 81900, loss: 30.102828979492188
epoch: 1, iter: 82000, loss: 30.472938537597656
epoch: 1, iteration: 82000, simlex-999: SpearmanrResult(correlation=0.162995795015982, pvalue=3.984582727922843e-07), men: SpearmanrResult(correlation=0.16521711507501075, pvalue=2.5829343239938948e-17), sim353: SpearmanrResult(correlation=0.2553242503620664, pvalue=3.9916456335010605e-06), nearest to monster: ['monster', 'triangle', 'giant', 'clown', 'robot', 'pen', 'cow', 'killer', 'bird', 'hammer']

epoch: 1, iter: 82100, loss: 30.87786102294922
epoch: 1, iter: 82200, loss: 30.369300842285156
epoch: 1, iter: 82300, loss: 30.81732749938965
epoch: 1, iter: 82400, loss: 30.576255798339844
epoch: 1, iter: 82500, loss: 30.190576553344727
epoch: 1, iter: 82600, loss: 30.46633529663086
epoch: 1, iter: 82700, loss: 30.395225524902344
epoch: 1, iter: 82800, loss: 30.413116455078125
epoch: 1, iter: 82900, loss: 30.11839485168457
epoch: 1, iter: 83000, loss: 30.749446868896484
epoch: 1, iter: 83100, loss: 30.579679489135742
epoch: 1, iter: 83200, loss: 30.242351531982422
epoch: 1, iter: 83300, loss: 30.643089294433594
epoch: 1, iter: 83400, loss: 31.025341033935547
epoch: 1, iter: 83500, loss: 30.11600112915039
epoch: 1, iter: 83600, loss: 30.297754287719727
epoch: 1, iter: 83700, loss: 30.28591537475586
epoch: 1, iter: 83800, loss: 30.40618896484375
epoch: 1, iter: 83900, loss: 30.353286743164062
epoch: 1, iter: 84000, loss: 29.93646240234375
epoch: 1, iteration: 84000, simlex-999: SpearmanrResult(correlation=0.16359501384641933, pvalue=3.6074490269985206e-07), men: SpearmanrResult(correlation=0.16651686883970906, pvalue=1.4439073198647016e-17), sim353: SpearmanrResult(correlation=0.2544400123109962, pvalue=4.321323577205728e-06), nearest to monster: ['monster', 'triangle', 'clown', 'robot', 'killer', 'giant', 'pen', 'hammer', 'bull', 'cow']

epoch: 1, iter: 84100, loss: 30.29789161682129
epoch: 1, iter: 84200, loss: 30.246540069580078
epoch: 1, iter: 84300, loss: 30.901538848876953
epoch: 1, iter: 84400, loss: 30.117076873779297
epoch: 1, iter: 84500, loss: 30.69649887084961
epoch: 1, iter: 84600, loss: 30.63212776184082
epoch: 1, iter: 84700, loss: 29.883480072021484
epoch: 1, iter: 84800, loss: 30.262691497802734
epoch: 1, iter: 84900, loss: 29.80817413330078
epoch: 1, iter: 85000, loss: 30.048599243164062
epoch: 1, iter: 85100, loss: 29.657894134521484
epoch: 1, iter: 85200, loss: 30.541133880615234
epoch: 1, iter: 85300, loss: 29.884437561035156
epoch: 1, iter: 85400, loss: 29.609167098999023
epoch: 1, iter: 85500, loss: 30.771604537963867
epoch: 1, iter: 85600, loss: 30.919052124023438
epoch: 1, iter: 85700, loss: 30.89596939086914
epoch: 1, iter: 85800, loss: 30.242305755615234
epoch: 1, iter: 85900, loss: 30.13596534729004
epoch: 1, iter: 86000, loss: 30.44934844970703
epoch: 1, iteration: 86000, simlex-999: SpearmanrResult(correlation=0.1631675148083149, pvalue=3.8727894368921164e-07), men: SpearmanrResult(correlation=0.1671210510930885, pvalue=1.1001130410358958e-17), sim353: SpearmanrResult(correlation=0.25503748920366215, pvalue=4.095839170272771e-06), nearest to monster: ['monster', 'triangle', 'clown', 'robot', 'killer', 'giant', 'bull', 'pen', 'cow', 'demon']

epoch: 1, iter: 86100, loss: 30.831403732299805
epoch: 1, iter: 86200, loss: 30.484277725219727
epoch: 1, iter: 86300, loss: 30.60747718811035
epoch: 1, iter: 86400, loss: 30.155363082885742
epoch: 1, iter: 86500, loss: 30.28110122680664
epoch: 1, iter: 86600, loss: 30.374900817871094
epoch: 1, iter: 86700, loss: 30.804969787597656
epoch: 1, iter: 86800, loss: 30.20755958557129
epoch: 1, iter: 86900, loss: 30.167919158935547
epoch: 1, iter: 87000, loss: 30.547744750976562
epoch: 1, iter: 87100, loss: 30.687185287475586
epoch: 1, iter: 87200, loss: 30.32683563232422
epoch: 1, iter: 87300, loss: 30.641101837158203
epoch: 1, iter: 87400, loss: 30.987831115722656
epoch: 1, iter: 87500, loss: 30.438377380371094
epoch: 1, iter: 87600, loss: 30.0216007232666
epoch: 1, iter: 87700, loss: 30.663925170898438
epoch: 1, iter: 87800, loss: 30.71135711669922
epoch: 1, iter: 87900, loss: 30.71870994567871
epoch: 1, iter: 88000, loss: 30.205699920654297
epoch: 1, iteration: 88000, simlex-999: SpearmanrResult(correlation=0.16425151824623327, pvalue=3.233772330038919e-07), men: SpearmanrResult(correlation=0.1685696359455271, pvalue=5.707930050658737e-18), sim353: SpearmanrResult(correlation=0.2556407027221877, pvalue=3.87959947193494e-06), nearest to monster: ['monster', 'triangle', 'giant', 'clown', 'bull', 'robot', 'killer', 'pen', 'demon', 'hammer']

epoch: 1, iter: 88100, loss: 30.927677154541016
epoch: 1, iter: 88200, loss: 30.55514144897461
epoch: 1, iter: 88300, loss: 30.23815155029297
epoch: 1, iter: 88400, loss: 30.708179473876953
epoch: 1, iter: 88500, loss: 30.492023468017578
epoch: 1, iter: 88600, loss: 30.046836853027344
epoch: 1, iter: 88700, loss: 29.993240356445312
epoch: 1, iter: 88800, loss: 30.803783416748047
epoch: 1, iter: 88900, loss: 30.327960968017578
epoch: 1, iter: 89000, loss: 29.940567016601562
epoch: 1, iter: 89100, loss: 30.143478393554688
epoch: 1, iter: 89200, loss: 31.123132705688477
epoch: 1, iter: 89300, loss: 30.397336959838867
epoch: 1, iter: 89400, loss: 30.187965393066406
epoch: 1, iter: 89500, loss: 30.048961639404297
epoch: 1, iter: 89600, loss: 30.07036590576172
epoch: 1, iter: 89700, loss: 30.79546356201172
epoch: 1, iter: 89800, loss: 29.865171432495117
epoch: 1, iter: 89900, loss: 30.40514373779297
epoch: 1, iter: 90000, loss: 30.69794464111328
epoch: 1, iteration: 90000, simlex-999: SpearmanrResult(correlation=0.16514616029140247, pvalue=2.784127116501988e-07), men: SpearmanrResult(correlation=0.16954415194873998, pvalue=3.658872001083876e-18), sim353: SpearmanrResult(correlation=0.2555836069543828, pvalue=3.899590976356642e-06), nearest to monster: ['monster', 'triangle', 'giant', 'bull', 'clown', 'robot', 'killer', 'hammer', 'cow', 'demon']

epoch: 1, iter: 90100, loss: 30.630199432373047
epoch: 1, iter: 90200, loss: 30.46442222595215
epoch: 1, iter: 90300, loss: 30.596553802490234
epoch: 1, iter: 90400, loss: 30.49251937866211
epoch: 1, iter: 90500, loss: 30.104934692382812
epoch: 1, iter: 90600, loss: 29.840747833251953
epoch: 1, iter: 90700, loss: 30.267772674560547
epoch: 1, iter: 90800, loss: 30.190765380859375
epoch: 1, iter: 90900, loss: 30.228517532348633
epoch: 1, iter: 91000, loss: 29.798009872436523
epoch: 1, iter: 91100, loss: 30.7586669921875
epoch: 1, iter: 91200, loss: 30.214977264404297
epoch: 1, iter: 91300, loss: 30.565635681152344
epoch: 1, iter: 91400, loss: 30.357879638671875
epoch: 1, iter: 91500, loss: 30.263957977294922
epoch: 1, iter: 91600, loss: 30.030994415283203
epoch: 1, iter: 91700, loss: 30.208396911621094
epoch: 1, iter: 91800, loss: 29.890247344970703
epoch: 1, iter: 91900, loss: 30.48164176940918
epoch: 1, iter: 92000, loss: 30.354490280151367
epoch: 1, iteration: 92000, simlex-999: SpearmanrResult(correlation=0.1653364321742003, pvalue=2.696593345422066e-07), men: SpearmanrResult(correlation=0.1688240673142005, pvalue=5.083536220386492e-18), sim353: SpearmanrResult(correlation=0.259105445573068, pvalue=2.8336170297537425e-06), nearest to monster: ['monster', 'giant', 'triangle', 'clown', 'robot', 'bull', 'cow', 'demon', 'reed', 'rod']

epoch: 1, iter: 92100, loss: 30.68956756591797
epoch: 1, iter: 92200, loss: 30.422218322753906
epoch: 1, iter: 92300, loss: 30.578102111816406
epoch: 1, iter: 92400, loss: 30.647659301757812
epoch: 1, iter: 92500, loss: 30.45465850830078
epoch: 1, iter: 92600, loss: 30.237232208251953
epoch: 1, iter: 92700, loss: 30.611480712890625
epoch: 1, iter: 92800, loss: 30.068378448486328
epoch: 1, iter: 92900, loss: 30.449426651000977
epoch: 1, iter: 93000, loss: 30.518421173095703
epoch: 1, iter: 93100, loss: 30.57355308532715
epoch: 1, iter: 93200, loss: 30.197509765625
epoch: 1, iter: 93300, loss: 30.10198974609375
epoch: 1, iter: 93400, loss: 30.406944274902344
epoch: 1, iter: 93500, loss: 30.485071182250977
epoch: 1, iter: 93600, loss: 30.394821166992188
epoch: 1, iter: 93700, loss: 30.108301162719727
epoch: 1, iter: 93800, loss: 30.055160522460938
epoch: 1, iter: 93900, loss: 30.503068923950195
epoch: 1, iter: 94000, loss: 30.17776870727539
epoch: 1, iteration: 94000, simlex-999: SpearmanrResult(correlation=0.16452175580014577, pvalue=3.091050503921563e-07), men: SpearmanrResult(correlation=0.17031641900147176, pvalue=2.5673219220449705e-18), sim353: SpearmanrResult(correlation=0.25769279178858767, pvalue=3.22261890503053e-06), nearest to monster: ['monster', 'giant', 'triangle', 'clown', 'robot', 'cow', 'bull', 'rod', 'reed', 'killer']

epoch: 1, iter: 94100, loss: 29.95470428466797
epoch: 1, iter: 94200, loss: 30.545040130615234
epoch: 1, iter: 94300, loss: 30.522565841674805
epoch: 1, iter: 94400, loss: 30.301044464111328
epoch: 1, iter: 94500, loss: 30.171085357666016
epoch: 1, iter: 94600, loss: 29.99677276611328
epoch: 1, iter: 94700, loss: 30.42511558532715
epoch: 1, iter: 94800, loss: 29.896169662475586
epoch: 1, iter: 94900, loss: 30.412710189819336
epoch: 1, iter: 95000, loss: 30.261638641357422
epoch: 1, iter: 95100, loss: 30.734485626220703
epoch: 1, iter: 95200, loss: 30.323421478271484
epoch: 1, iter: 95300, loss: 30.30504608154297
epoch: 1, iter: 95400, loss: 30.439306259155273
epoch: 1, iter: 95500, loss: 30.284433364868164
epoch: 1, iter: 95600, loss: 30.344554901123047
epoch: 1, iter: 95700, loss: 31.00472068786621
epoch: 1, iter: 95800, loss: 30.583446502685547
epoch: 1, iter: 95900, loss: 30.249595642089844
epoch: 1, iter: 96000, loss: 30.130508422851562
epoch: 1, iteration: 96000, simlex-999: SpearmanrResult(correlation=0.1658134194270542, pvalue=2.4886654321700024e-07), men: SpearmanrResult(correlation=0.16998066506649745, pvalue=2.9954700943620716e-18), sim353: SpearmanrResult(correlation=0.2615532866770961, pvalue=2.2634047025810973e-06), nearest to monster: ['monster', 'giant', 'triangle', 'clown', 'robot', 'cow', 'storyline', 'demon', 'killer', 'mine']

epoch: 1, iter: 96100, loss: 30.80952262878418
epoch: 1, iter: 96200, loss: 30.514129638671875
epoch: 1, iter: 96300, loss: 30.41751480102539
epoch: 1, iter: 96400, loss: 30.401996612548828
epoch: 1, iter: 96500, loss: 30.39910125732422
epoch: 1, iter: 96600, loss: 29.805984497070312
epoch: 1, iter: 96700, loss: 29.829578399658203
epoch: 1, iter: 96800, loss: 30.401958465576172
epoch: 1, iter: 96900, loss: 30.372657775878906
epoch: 1, iter: 97000, loss: 30.669193267822266
epoch: 1, iter: 97100, loss: 30.45154571533203
epoch: 1, iter: 97200, loss: 30.4871883392334
epoch: 1, iter: 97300, loss: 30.424468994140625
epoch: 1, iter: 97400, loss: 30.36046600341797
epoch: 1, iter: 97500, loss: 30.400753021240234
epoch: 1, iter: 97600, loss: 30.647533416748047
epoch: 1, iter: 97700, loss: 30.4141788482666
epoch: 1, iter: 97800, loss: 30.66451644897461
epoch: 1, iter: 97900, loss: 30.682424545288086
epoch: 1, iter: 98000, loss: 30.974720001220703
epoch: 1, iteration: 98000, simlex-999: SpearmanrResult(correlation=0.1656781876945656, pvalue=2.5459901947414803e-07), men: SpearmanrResult(correlation=0.17076454997186605, pvalue=2.0886359821470924e-18), sim353: SpearmanrResult(correlation=0.2634260651785924, pvalue=1.9030239792892506e-06), nearest to monster: ['monster', 'giant', 'triangle', 'robot', 'clown', 'cow', 'storyline', 'hammer', 'slogan', 'mine']

epoch: 1, iter: 98100, loss: 29.779621124267578
epoch: 1, iter: 98200, loss: 30.314851760864258
epoch: 1, iter: 98300, loss: 30.17748260498047
epoch: 1, iter: 98400, loss: 30.060449600219727
epoch: 1, iter: 98500, loss: 30.29900550842285
epoch: 1, iter: 98600, loss: 30.583925247192383
epoch: 1, iter: 98700, loss: 30.511886596679688
epoch: 1, iter: 98800, loss: 29.978679656982422
epoch: 1, iter: 98900, loss: 30.08024787902832
epoch: 1, iter: 99000, loss: 29.74579620361328
epoch: 1, iter: 99100, loss: 30.44879722595215
epoch: 1, iter: 99200, loss: 30.379261016845703
epoch: 1, iter: 99300, loss: 29.564411163330078
epoch: 1, iter: 99400, loss: 30.413551330566406
epoch: 1, iter: 99500, loss: 29.98810386657715
epoch: 1, iter: 99600, loss: 30.30841827392578
epoch: 1, iter: 99700, loss: 30.51578140258789
epoch: 1, iter: 99800, loss: 30.445234298706055
epoch: 1, iter: 99900, loss: 30.237821578979492
epoch: 1, iter: 100000, loss: 30.199050903320312
epoch: 1, iteration: 100000, simlex-999: SpearmanrResult(correlation=0.16715232584964468, pvalue=1.9843265854812782e-07), men: SpearmanrResult(correlation=0.17124287805386168, pvalue=1.6747013256150605e-18), sim353: SpearmanrResult(correlation=0.26460604437989393, pvalue=1.70487547849196e-06), nearest to monster: ['monster', 'giant', 'triangle', 'robot', 'clown', 'storyline', 'cow', 'slogan', 'hammer', 'melody']

epoch: 1, iter: 100100, loss: 30.333574295043945
epoch: 1, iter: 100200, loss: 30.55845832824707
epoch: 1, iter: 100300, loss: 30.75680160522461
epoch: 1, iter: 100400, loss: 30.388769149780273
epoch: 1, iter: 100500, loss: 30.058666229248047
epoch: 1, iter: 100600, loss: 30.498638153076172
epoch: 1, iter: 100700, loss: 30.53042984008789
epoch: 1, iter: 100800, loss: 30.348159790039062
epoch: 1, iter: 100900, loss: 30.638376235961914
epoch: 1, iter: 101000, loss: 30.172426223754883
epoch: 1, iter: 101100, loss: 30.896692276000977
epoch: 1, iter: 101200, loss: 30.448551177978516
epoch: 1, iter: 101300, loss: 30.56205177307129
epoch: 1, iter: 101400, loss: 30.40060043334961
epoch: 1, iter: 101500, loss: 30.31627655029297
epoch: 1, iter: 101600, loss: 30.167484283447266
epoch: 1, iter: 101700, loss: 30.0115966796875
epoch: 1, iter: 101800, loss: 30.624086380004883
epoch: 1, iter: 101900, loss: 30.172834396362305
epoch: 1, iter: 102000, loss: 30.562152862548828
epoch: 1, iteration: 102000, simlex-999: SpearmanrResult(correlation=0.1670230978748453, pvalue=2.028337490988046e-07), men: SpearmanrResult(correlation=0.1713810110701936, pvalue=1.5710283985855815e-18), sim353: SpearmanrResult(correlation=0.26441292634173025, pvalue=1.7358951566969633e-06), nearest to monster: ['monster', 'giant', 'triangle', 'robot', 'clown', 'storyline', 'cow', 'slogan', 'hammer', 'finger']

epoch: 1, iter: 102100, loss: 30.55624008178711
epoch: 1, iter: 102200, loss: 30.41084098815918
epoch: 1, iter: 102300, loss: 30.26647186279297
epoch: 1, iter: 102400, loss: 30.464323043823242
epoch: 1, iter: 102500, loss: 30.623504638671875
epoch: 1, iter: 102600, loss: 31.13153839111328
epoch: 1, iter: 102700, loss: 30.719829559326172
epoch: 1, iter: 102800, loss: 30.62751007080078
epoch: 1, iter: 102900, loss: 30.385581970214844
epoch: 1, iter: 103000, loss: 31.12250328063965
epoch: 1, iter: 103100, loss: 30.452640533447266
epoch: 1, iter: 103200, loss: 30.76857566833496
epoch: 1, iter: 103300, loss: 30.577342987060547
epoch: 1, iter: 103400, loss: 30.7468318939209
epoch: 1, iter: 103500, loss: 30.40059471130371
epoch: 1, iter: 103600, loss: 30.383655548095703
epoch: 1, iter: 103700, loss: 30.386371612548828
epoch: 1, iter: 103800, loss: 30.170000076293945
epoch: 1, iter: 103900, loss: 30.548282623291016
epoch: 1, iter: 104000, loss: 29.733184814453125
epoch: 1, iteration: 104000, simlex-999: SpearmanrResult(correlation=0.16893464777171635, pvalue=1.4637531128005816e-07), men: SpearmanrResult(correlation=0.17193637752975074, pvalue=1.2144154074333336e-18), sim353: SpearmanrResult(correlation=0.2644817771205538, pvalue=1.72477455415369e-06), nearest to monster: ['monster', 'giant', 'robot', 'triangle', 'clown', 'storyline', 'hammer', 'bull', 'slogan', 'cow']

epoch: 1, iter: 104100, loss: 30.98358917236328
epoch: 1, iter: 104200, loss: 30.25103759765625
epoch: 1, iter: 104300, loss: 30.568321228027344
epoch: 1, iter: 104400, loss: 30.607681274414062
epoch: 1, iter: 104500, loss: 31.158170700073242
epoch: 1, iter: 104600, loss: 30.96269416809082
epoch: 1, iter: 104700, loss: 30.154834747314453
epoch: 1, iter: 104800, loss: 30.38031768798828
epoch: 1, iter: 104900, loss: 30.168392181396484
epoch: 1, iter: 105000, loss: 30.190406799316406
epoch: 1, iter: 105100, loss: 30.59976577758789
epoch: 1, iter: 105200, loss: 30.05843734741211
epoch: 1, iter: 105300, loss: 30.54991340637207
epoch: 1, iter: 105400, loss: 30.098140716552734
epoch: 1, iter: 105500, loss: 30.414899826049805
epoch: 1, iter: 105600, loss: 30.54265594482422
epoch: 1, iter: 105700, loss: 30.9292049407959
epoch: 1, iter: 105800, loss: 30.35509490966797
epoch: 1, iter: 105900, loss: 30.05394172668457
epoch: 1, iter: 106000, loss: 30.226543426513672
epoch: 1, iteration: 106000, simlex-999: SpearmanrResult(correlation=0.1684798203405728, pvalue=1.5824218505055718e-07), men: SpearmanrResult(correlation=0.1734020098336969, pvalue=6.130244612215065e-19), sim353: SpearmanrResult(correlation=0.2653148902030666, pvalue=1.595499439172943e-06), nearest to monster: ['monster', 'giant', 'robot', 'clown', 'triangle', 'hammer', 'bull', 'storyline', 'slogan', 'cow']

epoch: 1, iter: 106100, loss: 30.15334701538086
epoch: 1, iter: 106200, loss: 30.41736602783203
epoch: 1, iter: 106300, loss: 30.16756820678711
epoch: 1, iter: 106400, loss: 30.476585388183594
epoch: 1, iter: 106500, loss: 31.16946792602539
epoch: 1, iter: 106600, loss: 30.209535598754883
epoch: 1, iter: 106700, loss: 30.48019790649414
epoch: 1, iter: 106800, loss: 29.56353187561035
epoch: 1, iter: 106900, loss: 30.35293960571289
epoch: 1, iter: 107000, loss: 30.54877471923828
epoch: 1, iter: 107100, loss: 29.95642852783203
epoch: 1, iter: 107200, loss: 29.863868713378906
epoch: 1, iter: 107300, loss: 30.54117202758789
epoch: 1, iter: 107400, loss: 30.564319610595703
epoch: 1, iter: 107500, loss: 30.289745330810547
epoch: 1, iter: 107600, loss: 30.34166717529297
epoch: 1, iter: 107700, loss: 30.228656768798828
epoch: 1, iter: 107800, loss: 30.19826889038086
epoch: 1, iter: 107900, loss: 29.98631477355957
epoch: 1, iter: 108000, loss: 30.534252166748047
epoch: 1, iteration: 108000, simlex-999: SpearmanrResult(correlation=0.16809445029314032, pvalue=1.6901904840218488e-07), men: SpearmanrResult(correlation=0.17424501907579965, pvalue=4.1259778445697578e-19), sim353: SpearmanrResult(correlation=0.2659931356800956, pvalue=1.4971527422559394e-06), nearest to monster: ['monster', 'giant', 'robot', 'hammer', 'triangle', 'bull', 'clown', 'rod', 'storyline', 'slogan']

epoch: 1, iter: 108100, loss: 30.33737564086914
epoch: 1, iter: 108200, loss: 30.28573226928711
epoch: 1, iter: 108300, loss: 30.758136749267578
epoch: 1, iter: 108400, loss: 30.248483657836914
epoch: 1, iter: 108500, loss: 29.876144409179688
epoch: 1, iter: 108600, loss: 30.282499313354492
epoch: 1, iter: 108700, loss: 30.403133392333984
epoch: 1, iter: 108800, loss: 30.222488403320312
epoch: 1, iter: 108900, loss: 30.056053161621094
epoch: 1, iter: 109000, loss: 30.39307403564453
epoch: 1, iter: 109100, loss: 30.315738677978516
epoch: 1, iter: 109200, loss: 30.37272071838379
epoch: 1, iter: 109300, loss: 30.462688446044922
epoch: 1, iter: 109400, loss: 30.42410659790039
epoch: 1, iter: 109500, loss: 30.414215087890625
epoch: 1, iter: 109600, loss: 30.320392608642578
epoch: 1, iter: 109700, loss: 30.297039031982422
epoch: 1, iter: 109800, loss: 30.644512176513672
epoch: 1, iter: 109900, loss: 30.753337860107422
epoch: 1, iter: 110000, loss: 30.265583038330078
epoch: 1, iteration: 110000, simlex-999: SpearmanrResult(correlation=0.16907060515792793, pvalue=1.4299814068693673e-07), men: SpearmanrResult(correlation=0.17510555147417187, pvalue=2.7485794885676823e-19), sim353: SpearmanrResult(correlation=0.267975254442813, pvalue=1.241886586688811e-06), nearest to monster: ['monster', 'giant', 'robot', 'bull', 'hammer', 'clown', 'rod', 'triangle', 'killer', 'reads']

epoch: 1, iter: 110100, loss: 30.272464752197266
epoch: 1, iter: 110200, loss: 30.38793182373047
epoch: 1, iter: 110300, loss: 30.590267181396484
epoch: 1, iter: 110400, loss: 30.97867202758789
epoch: 1, iter: 110500, loss: 30.195693969726562
epoch: 1, iter: 110600, loss: 30.050588607788086
epoch: 1, iter: 110700, loss: 30.010971069335938
epoch: 1, iter: 110800, loss: 30.200347900390625
epoch: 1, iter: 110900, loss: 30.716394424438477
epoch: 1, iter: 111000, loss: 30.02122688293457
epoch: 1, iter: 111100, loss: 30.24693489074707
epoch: 1, iter: 111200, loss: 30.085987091064453
epoch: 1, iter: 111300, loss: 30.499698638916016
epoch: 1, iter: 111400, loss: 30.532825469970703
epoch: 1, iter: 111500, loss: 29.860715866088867
epoch: 1, iter: 111600, loss: 30.18459701538086
epoch: 1, iter: 111700, loss: 30.063079833984375
epoch: 1, iter: 111800, loss: 30.4438533782959
epoch: 1, iter: 111900, loss: 29.979290008544922
epoch: 1, iter: 112000, loss: 29.959312438964844
epoch: 1, iteration: 112000, simlex-999: SpearmanrResult(correlation=0.17045668307885228, pvalue=1.1259417092002108e-07), men: SpearmanrResult(correlation=0.17566254497426181, pvalue=2.110750452163048e-19), sim353: SpearmanrResult(correlation=0.2669130119391746, pvalue=1.3729997266364257e-06), nearest to monster: ['monster', 'giant', 'bull', 'robot', 'clown', 'hammer', 'triangle', 'killer', 'reads', 'slogan']

epoch: 1, iter: 112100, loss: 30.033226013183594
epoch: 1, iter: 112200, loss: 29.982528686523438
epoch: 1, iter: 112300, loss: 30.576221466064453
epoch: 1, iter: 112400, loss: 30.227249145507812
epoch: 1, iter: 112500, loss: 30.463727951049805
epoch: 1, iter: 112600, loss: 30.63485336303711
epoch: 1, iter: 112700, loss: 30.5284423828125
epoch: 1, iter: 112800, loss: 30.665870666503906
epoch: 1, iter: 112900, loss: 29.814481735229492
epoch: 1, iter: 113000, loss: 30.23995590209961
epoch: 1, iter: 113100, loss: 30.29913330078125
epoch: 1, iter: 113200, loss: 30.609512329101562
epoch: 1, iter: 113300, loss: 30.66790771484375
epoch: 1, iter: 113400, loss: 30.355817794799805
epoch: 1, iter: 113500, loss: 30.777101516723633
epoch: 1, iter: 113600, loss: 29.784292221069336
epoch: 1, iter: 113700, loss: 30.44632339477539
epoch: 1, iter: 113800, loss: 30.35899543762207
epoch: 1, iter: 113900, loss: 30.091773986816406
epoch: 1, iter: 114000, loss: 30.235483169555664
epoch: 1, iteration: 114000, simlex-999: SpearmanrResult(correlation=0.17074223750408546, pvalue=1.0715770710689782e-07), men: SpearmanrResult(correlation=0.176259639644225, pvalue=1.5888720988857355e-19), sim353: SpearmanrResult(correlation=0.2676783937677228, pvalue=1.2772687113870497e-06), nearest to monster: ['monster', 'giant', 'bull', 'robot', 'clown', 'hammer', 'killer', 'triangle', 'slogan', 'storyline']

epoch: 1, iter: 114100, loss: 31.208988189697266
epoch: 1, iter: 114200, loss: 30.55046844482422
epoch: 1, iter: 114300, loss: 29.999637603759766
epoch: 1, iter: 114400, loss: 29.898786544799805
epoch: 1, iter: 114500, loss: 30.682052612304688
epoch: 1, iter: 114600, loss: 30.83867835998535
epoch: 1, iter: 114700, loss: 30.36109733581543
epoch: 1, iter: 114800, loss: 30.97061538696289
epoch: 1, iter: 114900, loss: 30.502185821533203
epoch: 1, iter: 115000, loss: 30.31426239013672
epoch: 1, iter: 115100, loss: 30.291278839111328
epoch: 1, iter: 115200, loss: 29.921966552734375
epoch: 1, iter: 115300, loss: 30.482467651367188
epoch: 1, iter: 115400, loss: 30.56399917602539
epoch: 1, iter: 115500, loss: 30.21322250366211
epoch: 1, iter: 115600, loss: 30.300979614257812
epoch: 1, iter: 115700, loss: 30.243253707885742
epoch: 1, iter: 115800, loss: 30.047298431396484
epoch: 1, iter: 115900, loss: 30.42316436767578
epoch: 1, iter: 116000, loss: 30.36935043334961
epoch: 1, iteration: 116000, simlex-999: SpearmanrResult(correlation=0.17049890390424202, pvalue=1.1177389455317111e-07), men: SpearmanrResult(correlation=0.1761607002941838, pvalue=1.6655510704355566e-19), sim353: SpearmanrResult(correlation=0.26764518119690817, pvalue=1.2812867355896495e-06), nearest to monster: ['monster', 'giant', 'bull', 'clown', 'robot', 'hammer', 'killer', 'triangle', 'vampire', 'demon']

epoch: 1, iter: 116100, loss: 29.956588745117188
epoch: 1, iter: 116200, loss: 30.239765167236328
epoch: 1, iter: 116300, loss: 30.056724548339844
epoch: 1, iter: 116400, loss: 30.337177276611328
epoch: 1, iter: 116500, loss: 30.554336547851562
epoch: 1, iter: 116600, loss: 30.801679611206055
epoch: 1, iter: 116700, loss: 30.705276489257812
epoch: 1, iter: 116800, loss: 30.503780364990234
epoch: 1, iter: 116900, loss: 29.62342643737793
epoch: 1, iter: 117000, loss: 30.29004669189453
epoch: 1, iter: 117100, loss: 30.506996154785156
epoch: 1, iter: 117200, loss: 30.6331787109375
epoch: 1, iter: 117300, loss: 30.65314483642578
epoch: 1, iter: 117400, loss: 30.795137405395508
epoch: 1, iter: 117500, loss: 30.28030776977539
epoch: 1, iter: 117600, loss: 30.351322174072266
epoch: 1, iter: 117700, loss: 30.542030334472656
epoch: 1, iter: 117800, loss: 30.361120223999023
epoch: 1, iter: 117900, loss: 30.456024169921875
epoch: 1, iter: 118000, loss: 30.537174224853516
epoch: 1, iteration: 118000, simlex-999: SpearmanrResult(correlation=0.17280547641221922, pvalue=7.475906386705914e-08), men: SpearmanrResult(correlation=0.17732985405232565, pvalue=9.526064095701539e-20), sim353: SpearmanrResult(correlation=0.2698694905041053, pvalue=1.037261623872224e-06), nearest to monster: ['monster', 'giant', 'robot', 'clown', 'hammer', 'bull', 'killer', 'vampire', 'triangle', 'demon']

epoch: 1, iter: 118100, loss: 29.840484619140625
epoch: 1, iter: 118200, loss: 30.325119018554688
epoch: 1, iter: 118300, loss: 30.51473045349121
epoch: 1, iter: 118400, loss: 30.261699676513672
epoch: 1, iter: 118500, loss: 30.180068969726562
epoch: 1, iter: 118600, loss: 30.017879486083984
epoch: 1, iter: 118700, loss: 30.56424903869629
epoch: 1, iter: 118800, loss: 30.457590103149414
epoch: 1, iter: 118900, loss: 30.63213539123535
epoch: 1, iter: 119000, loss: 30.692546844482422
epoch: 1, iter: 119100, loss: 30.539554595947266
epoch: 1, iter: 119200, loss: 30.656726837158203
epoch: 1, iter: 119300, loss: 30.380685806274414
epoch: 1, iter: 119400, loss: 29.897314071655273
epoch: 1, iter: 119500, loss: 29.90090560913086

1	model.load_state_dict(torch.load("embedding-{}.th".format(EMBEDDING_SIZE)))

在 MEN 和 Simplex-999 数据集上做评估

embedding_weights = model.input_embeddings()
print("simlex-999", evaluate("simlex-999.txt", embedding_weights))
print("men", evaluate("men.txt", embedding_weights))
print("wordsim353", evaluate("wordsim353.csv", embedding_weights))

simlex-999 SpearmanrResult(correlation=0.17251697429101504, pvalue=7.863946056740345e-08)
men SpearmanrResult(correlation=0.1778096817088841, pvalue=7.565661657312768e-20)
wordsim353 SpearmanrResult(correlation=0.27153702278146635, pvalue=8.842165885381714e-07)

寻找nearest neighbors

1 2	for word in ["good", "fresh", "monster", "green", "like", "america", "chicago", "work", "computer", "language"]: print(word, find_nearest(word))

good ['good', 'bad', 'perfect', 'hard', 'questions', 'alone', 'money', 'false', 'truth', 'experience']
fresh ['fresh', 'grain', 'waste', 'cooling', 'lighter', 'dense', 'mild', 'sized', 'warm', 'steel']
monster ['monster', 'giant', 'robot', 'hammer', 'clown', 'bull', 'demon', 'triangle', 'storyline', 'slogan']
green ['green', 'blue', 'yellow', 'white', 'cross', 'orange', 'black', 'red', 'mountain', 'gold']
like ['like', 'unlike', 'etc', 'whereas', 'animals', 'soft', 'amongst', 'similarly', 'bear', 'drink']
america ['america', 'africa', 'korea', 'india', 'australia', 'turkey', 'pakistan', 'mexico', 'argentina', 'carolina']
chicago ['chicago', 'boston', 'illinois', 'texas', 'london', 'indiana', 'massachusetts', 'florida', 'berkeley', 'michigan']
work ['work', 'writing', 'job', 'marx', 'solo', 'label', 'recording', 'nietzsche', 'appearance', 'stage']
computer ['computer', 'digital', 'electronic', 'audio', 'video', 'graphics', 'hardware', 'software', 'computers', 'program']
language ['language', 'languages', 'alphabet', 'arabic', 'grammar', 'pronunciation', 'dialect', 'programming', 'chinese', 'spelling']

单词之间的关系

man_idx = word_to_idx["man"] 
king_idx = word_to_idx["king"] 
woman_idx = word_to_idx["woman"]
embedding = embedding_weights[woman_idx] - embedding_weights[man_idx] + embedding_weights[king_idx]
cos_dis = np.array([scipy.spatial.distance.cosine(e, embedding) for e in embedding_weights])
for i in cos_dis.argsort()[:20]:
    print(idx_to_word[i])

king
henry
charles
pope
queen
iii
prince
elizabeth
alexander
constantine
edward
son
iv
louis
emperor
mary
james
joseph
frederick
francis

Hello World

2020-02-29

Welcome to Hexo! This is your very first post. Check documentation for more info. If you get any problems when using Hexo, you can find the answer in troubleshooting or you can ask me on GitHub.

Quick Start

Create a new post

1	$ hexo new "My New Post"

More info: Writing

Run server

1	$ hexo server

More info: Server

Generate static files

1	$ hexo generate

More info: Generating

Deploy to remote sites

1	$ hexo deploy

More info: Deployment

My New Post

2020-02-28

PyTorch入门与实战第六课

目录

图片风格迁移 Neural Style Transfer

Generative Adversarial Networks

DCGAN

第五课 CNN图像分类

CNN模型的迁移学习

数据

读入数据

课后学习

第一课

什么是PyTorch?

Tensors

注意

注意

Numpy和Tensor之间的转化

CUDA Tensors

热身: 用numpy实现两层神经网络

PyTorch: Tensors

PyTorch: Tensor和autograd

PyTorch: nn

PyTorch: optim

PyTorch: 自定义 nn Modules

FizzBuzz

第六课 Seq2Seq, Attention

更多阅读

课件

论文

PyTorch代码

更多关于Machine Translation

TODO

NER

没有Attention的版本

Encoder

Luong Attention

Decoder

Seq2Seq

情感分析

准备数据

Word Averaging模型

训练模型

RNN模型

训练RNN模型

我们来尝试一个AVG的版本

训练RNN模型

CNN模型

第四课 情感分析

准备数据

Word Averaging模型

训练模型

RNN模型

训练RNN模型

CNN模型

第三课 语言模型

定义模型

使用最好的模型在valid数据上计算perplexity

使用最好的模型在测试数据上计算perplexity

第二课 词向量

实现Dataloader

定义PyTorch模型

在 MEN 和 Simplex-999 数据集上做评估

寻找nearest neighbors

单词之间的关系

Quick Start

Create a new post

Run server

Generate static files

Deploy to remote sites

第四课情感分析

第三课语言模型

第二课词向量