Linear solvers in the TT format

This tutorial addresses solving multilinear systems \(\mathsf{Ax}=\mathsf{b}\) in the TT format.

Imports:

[1]:

import torch as tn import datetime try: import torchtt as tntt except: print('Installing torchTT...') %pip install git+https://github.com/ion-g-ion/torchTT import torchtt as tntt

Small example

A random tensor operator \(\mathsf{A}\) is created in the TT format. We create a random right-hand side \(\mathsf{b} = \mathsf{Ax}\), where \(\mathsf{x}\) is a random tensor in the TT format. This way the solution of \(\mathsf{Ax}=\mathsf{b}\) is known and we can compare it as a reference. This works only for small random tensors.

[2]:
A = tntt.random([(4,4),(5,5),(6,6)],[1,2,3,1])
x = tntt.random([4,5,6],[1,2,3,1])
b = A @ x

Solve the multilinear system \(\mathsf{Ax}=\mathsf{b}\) using the method torchtt.solvers.amen_solve().

[3]:
xs = tntt.solvers.amen_solve(A,b, x0 = b, eps = 1e-7)

The relative residual norm and the relative error of the solution are reported:

[4]:
print(xs)
print('Relative residual error ',(A@xs-b).norm()/b.norm())
print('Relative error of the solution  ',(xs-x).norm()/x.norm())

TT with sizes and ranks:
N = [4, 5, 6]
R = [1, 2, 3, 1]

Device: cpu, dtype: torch.float64
#entries 56 compression 0.4666666666666667

Relative residual error  tensor(4.7719e-10, dtype=torch.float64)
Relative error of the solution   tensor(4.9390e-10, dtype=torch.float64)

Finite differences

We now solve the problem \(\Delta u = 1\) in \([0,1]^d\) with $ u = 0 $ on the entire boundary using finite differences. First, set the size of the problem (n is the mode size and d is the number of dimensions):

[5]:
dtype = tn.float64
n =  256
d = 8

Create the finite differences matrix corresponding to the problem. The operator is constructed directly in the TT format as it follows

[6]:
L1d = -2*tn.eye(n, dtype = dtype)+tn.diag(tn.ones(n-1,dtype = dtype),-1)+tn.diag(tn.ones(n-1,dtype = dtype),1)
L1d[0,1] = 0
L1d[-1,-2] = 0
L1d /= (n-1)
L1d = tntt.TT(L1d, [(n,n)])

L_tt = tntt.zeros([(n,n)]*d)
for i in range(1,d-1):
    L_tt = L_tt+tntt.eye([n]*i)**L1d**tntt.eye([n]*(d-1-i))
L_tt = L_tt + L1d**tntt.eye([n]*(d-1)) +  tntt.eye([n]*(d-1))**L1d
L_tt = L_tt.round(1e-14)

The right hand site of the finite difference system is also computed in the TT format

[7]:
b1d = tn.ones(n, dtype=dtype)
b1d[0] = 0
b1d[-1] = 0
b1d = tntt.TT(b1d)
b_tt = b1d
for i in range(d-1):
    b_tt = b_tt**b1d

Solve the system

[8]:
time = datetime.datetime.now()
x = tntt.solvers.amen_solve(L_tt, b_tt ,x0 = b_tt, nswp = 20, eps = 1e-8, verbose = True)
time = datetime.datetime.now() - time
print('Relative residual: ',(L_tt@x-b_tt).norm()/b_tt.norm())
print('Solver time: ',time)
Starting AMEn solve with:
        epsilon              : 1e-08
        sweeps               : 20
        local iterations     : 40
        resets               : 2
        local preconditioner : N


Starting sweep 1
        Core 0
                Choosing direct solver (local size 256)...
                dx = 1.00022, res_now = 3.94265e-15, res_old = 1.00025
        Core 1
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 2.64902e-07 (from 1.12368e-09)
                Time needed 24.064 ms
                dx = 0.162254, res_now = 2.64902e-07, res_old = 1.5732
        Core 2
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 6.70485e-07 (from 9.82742e-10)
                Time needed 38.405 ms
                dx = 0.178782, res_now = 6.70485e-07, res_old = 1.79881
        Core 3
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 2.47626e-06 (from 8.421e-10)
                Time needed 48.229 ms
                dx = 0.201381, res_now = 2.47626e-06, res_old = 2.09924
        Core 4
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 9.56499e-06 (from 6.99836e-10)
                Time needed 48.069 ms
                dx = 0.23409, res_now = 9.56499e-06, res_old = 2.52597
        Core 5
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 5.18408e-05 (from 5.56264e-10)
                Time needed 50.732 ms
                dx = 0.285484, res_now = 5.18408e-05, res_old = 3.17793
        Core 6
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.000422745 (from 4.10681e-10)
                Time needed 38.09 ms
                dx = 0.376877, res_now = 0.000422745, res_old = 4.30448
        Core 7
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.00923264 (from 2.6303e-10)
                Time needed 21.29 ms
                dx = 0.579017, res_now = 0.00923264, res_old = 6.72078
Solution rank [ 1 5 5 5 5 5 5 5 1 ]
Maxres 6.72078
Time 295.645 ms

Starting sweep 2
        Core 0
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.015374 (from 1.32307e-09)
                Time needed 21.018 ms
                dx = 0.172509, res_now = 0.015374, res_old = 1.33611
        Core 1
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0343199 (from 7.96653e-10)
                Time needed 115.429 ms
                dx = 0.168754, res_now = 0.0343199, res_old = 2.21899
        Core 2
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0538122 (from 6.91437e-10)
                Time needed 114.713 ms
                dx = 0.156787, res_now = 0.0538122, res_old = 2.55666
        Core 3
                Choosing iterative solver (local size 7680)...
                Finished with flag 0 after 80 iterations with relres 0.0463301 (from 6.83955e-10)
                Time needed 121.182 ms
                dx = 0.17905, res_now = 0.0463301, res_old = 2.58462
        Core 4
                Choosing iterative solver (local size 7680)...
                Finished with flag 0 after 80 iterations with relres 0.0497495 (from 6.9982e-10)
                Time needed 125.369 ms
                dx = 0.172615, res_now = 0.0497495, res_old = 2.52603
        Core 5
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0681063 (from 7.62661e-10)
                Time needed 121.283 ms
                dx = 0.147447, res_now = 0.0681063, res_old = 2.31789
        Core 6
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0809247 (from 1.42434e-09)
                Time needed 113.943 ms
                dx = 0.129625, res_now = 0.0809247, res_old = 1.24111
        Core 7
                Choosing iterative solver (local size 1792)...
                Finished with flag 0 after 80 iterations with relres 0.042695 (from 3.48983e-09)
                Time needed 25.323 ms
                dx = 0.0939197, res_now = 0.042695, res_old = 0.506549
Solution rank [ 1 9 9 6 6 9 9 7 1 ]
Maxres 2.58462
Time 806.999 ms

Starting sweep 3
        Core 0
                Choosing iterative solver (local size 2304)...
                Finished with flag 0 after 80 iterations with relres 0.03434 (from 2.69626e-09)
                Time needed 24.868 ms
                dx = 0.0822758, res_now = 0.03434, res_old = 0.655636
        Core 1
                Choosing iterative solver (local size 29952)...
                Finished with flag 0 after 80 iterations with relres 0.0270942 (from 3.39634e-09)
                Time needed 134.154 ms
                dx = 0.0526669, res_now = 0.0270942, res_old = 0.520491
        Core 2
                Choosing iterative solver (local size 13824)...
                Finished with flag 0 after 80 iterations with relres 0.0231415 (from 3.05189e-09)
                Time needed 103.604 ms
                dx = 0.0537463, res_now = 0.0231415, res_old = 0.579238
        Core 3
                Choosing iterative solver (local size 15360)...
                Finished with flag 0 after 80 iterations with relres 0.0224298 (from 2.06727e-09)
                Time needed 112.814 ms
                dx = 0.0430813, res_now = 0.0224298, res_old = 0.855123
        Core 4
                Choosing iterative solver (local size 18432)...
                Finished with flag 0 after 80 iterations with relres 0.0193917 (from 4.70433e-09)
                Time needed 143.791 ms
                dx = 0.0372849, res_now = 0.0193917, res_old = 0.375775
        Core 5
                Choosing iterative solver (local size 18432)...
                Finished with flag 0 after 80 iterations with relres 0.0164225 (from 4.13588e-09)
                Time needed 134.821 ms
                dx = 0.046789, res_now = 0.0164225, res_old = 0.427422
        Core 6
                Choosing iterative solver (local size 14336)...
                Finished with flag 0 after 80 iterations with relres 0.0164913 (from 4.01682e-09)
                Time needed 105.785 ms
                dx = 0.0483424, res_now = 0.0164913, res_old = 0.440091
        Core 7
                Choosing iterative solver (local size 2816)...
                Finished with flag 0 after 80 iterations with relres 0.00957893 (from 2.5338e-08)
                Time needed 24.291 ms
                dx = 0.028046, res_now = 0.00957893, res_old = 0.0697674
Solution rank [ 1 13 9 10 8 8 8 11 1 ]
Maxres 0.855123
Time 843.258 ms

Starting sweep 4
        Core 0
                Choosing iterative solver (local size 3328)...
                Finished with flag 0 after 80 iterations with relres 0.00517621 (from 1.77343e-08)
                Time needed 21.536 ms
                dx = 0.0151357, res_now = 0.00517621, res_old = 0.0996806
        Core 1
                Choosing iterative solver (local size 23040)...
                Finished with flag 0 after 80 iterations with relres 0.00573939 (from 1.60449e-08)
                Time needed 113.317 ms
                dx = 0.0132804, res_now = 0.00573939, res_old = 0.110176
        Core 2
                Choosing iterative solver (local size 28160)...
                Finished with flag 0 after 80 iterations with relres 0.00406291 (from 1.81947e-08)
                Time needed 156.767 ms
                dx = 0.0104602, res_now = 0.00406291, res_old = 0.0971585
        Core 3
                Choosing iterative solver (local size 20480)...
                Finished with flag 0 after 80 iterations with relres 0.00365042 (from 2.80398e-08)
                Time needed 144.82 ms
                dx = 0.0122825, res_now = 0.00365042, res_old = 0.063045
        Core 4
                Choosing iterative solver (local size 24576)...
                Finished with flag 0 after 80 iterations with relres 0.00390755 (from 2.65709e-08)
                Time needed 158.724 ms
                dx = 0.00831183, res_now = 0.00390755, res_old = 0.0665301
        Core 5
                Choosing iterative solver (local size 22528)...
                Finished with flag 0 after 80 iterations with relres 0.00263752 (from 2.66631e-08)
                Time needed 161.952 ms
                dx = 0.00624501, res_now = 0.00263752, res_old = 0.0663
        Core 6
                Choosing iterative solver (local size 30976)...
                Finished with flag 0 after 80 iterations with relres 0.00213413 (from 2.64413e-08)
                Time needed 158.544 ms
                dx = 0.00564083, res_now = 0.00213413, res_old = 0.0668564
        Core 7
                Choosing iterative solver (local size 3840)...
                Finished with flag 0 after 80 iterations with relres 0.000777821 (from 2.04659e-07)
                Time needed 26.838 ms
                dx = 0.00151834, res_now = 0.000777821, res_old = 0.00863764
Solution rank [ 1 10 11 10 12 11 11 15 1 ]
Maxres 0.110176
Time 1003.79 ms

Starting sweep 5
        Core 0
                Choosing iterative solver (local size 2560)...
                Finished with flag 0 after 80 iterations with relres 0.000687232 (from 1.19226e-07)
                Time needed 21.055 ms
                dx = 0.00266233, res_now = 0.000687232, res_old = 0.0148271
        Core 1
                Choosing iterative solver (local size 39424)...
                Finished with flag 0 after 80 iterations with relres 0.000495394 (from 7.83787e-08)
                Time needed 128.263 ms
                dx = 0.00119648, res_now = 0.000495394, res_old = 0.0225542
        Core 2
                Choosing iterative solver (local size 35840)...
                Finished with flag 0 after 80 iterations with relres 0.000314899 (from 1.23665e-07)
                Time needed 158.242 ms
                dx = 0.00069244, res_now = 0.000314899, res_old = 0.0142948
        Core 3
                Choosing iterative solver (local size 43008)...
                Finished with flag 0 after 80 iterations with relres 0.000341951 (from 9.01584e-08)
                Time needed 210.742 ms
                dx = 0.000653499, res_now = 0.000341951, res_old = 0.0196073
        Core 4
                Choosing iterative solver (local size 45056)...
                Finished with flag 0 after 80 iterations with relres 0.00048883 (from 9.50063e-08)
                Time needed 212.996 ms
                dx = 0.00117568, res_now = 0.00048883, res_old = 0.0186068
        Core 5
                Choosing iterative solver (local size 39424)...
                Finished with flag 0 after 80 iterations with relres 0.000298197 (from 2.28206e-07)
                Time needed 222.749 ms
                dx = 0.000646952, res_now = 0.000298197, res_old = 0.00774638
        Core 6
                Choosing iterative solver (local size 53760)...
                Finished with flag 0 after 80 iterations with relres 0.000208802 (from 3.57617e-07)
                Time needed 239.754 ms
                dx = 0.000534198, res_now = 0.000208802, res_old = 0.00494319
        Core 7
                Choosing iterative solver (local size 3840)...
                Finished with flag 0 after 80 iterations with relres 2.68295e-05 (from 4.01085e-06)
                Time needed 34.463 ms
                dx = 0.000112321, res_now = 2.68295e-05, res_old = 0.000440746
Solution rank [ 1 14 14 14 16 14 14 15 1 ]
Maxres 0.0225542
Time 1308.83 ms

Starting sweep 6
        Core 0
                Choosing iterative solver (local size 3584)...
                Finished with flag 0 after 80 iterations with relres 5.55609e-05 (from 1.09622e-06)
                Time needed 23.115 ms
                dx = 0.000170061, res_now = 5.55609e-05, res_old = 0.0016126
        Core 1
                Choosing iterative solver (local size 53760)...
                Finished with flag 0 after 80 iterations with relres 5.62707e-05 (from 1.1471e-06)
                Time needed 155.1 ms
                dx = 0.000109019, res_now = 5.62707e-05, res_old = 0.00154108
        Core 2
                Choosing iterative solver (local size 60928)...
                Finished with flag 0 after 80 iterations with relres 5.67046e-05 (from 8.53007e-07)
                Time needed 226.331 ms
                dx = 0.000121998, res_now = 5.67046e-05, res_old = 0.0020724
        Core 3
                Choosing iterative solver (local size 73728)...
                Finished with flag 0 after 80 iterations with relres 4.17311e-05 (from 9.9292e-07)
                Time needed 305.458 ms
                dx = 0.00011031, res_now = 4.17311e-05, res_old = 0.00178037
        Core 4
                Choosing iterative solver (local size 60928)...
                Finished with flag 0 after 80 iterations with relres 3.06985e-05 (from 1.12449e-06)
                Time needed 259.442 ms
                dx = 7.89894e-05, res_now = 3.06985e-05, res_old = 0.00157206
        Core 5
                Choosing iterative solver (local size 57344)...
                Finished with flag 0 after 80 iterations with relres 3.21308e-05 (from 2.12004e-06)
                Time needed 255.088 ms
                dx = 8.46829e-05, res_now = 3.21308e-05, res_old = 0.000833835
        Core 6
                Choosing iterative solver (local size 69120)...
                Finished with flag 0 after 80 iterations with relres 1.10649e-05 (from 4.29914e-06)
                Time needed 216.055 ms
                dx = 2.29139e-05, res_now = 1.10649e-05, res_old = 0.000411191
        Core 7
                Choosing iterative solver (local size 4864)...
                Finished with flag 0 after 80 iterations with relres 2.65671e-06 (from 4.96271e-05)
                Time needed 63.156 ms
                dx = 6.78001e-06, res_now = 2.65671e-06, res_old = 3.5621e-05
Solution rank [ 1 15 17 18 17 16 18 19 1 ]
Maxres 0.0020724
Time 1600.99 ms

Starting sweep 7
        Core 0
                Choosing iterative solver (local size 3840)...
                Finished with flag 0 after 80 iterations with relres 2.55923e-06 (from 1.88144e-05)
                Time needed 23.444 ms
                dx = 1.24458e-05, res_now = 2.55923e-06, res_old = 9.39583e-05
        Core 1
                Choosing iterative solver (local size 73984)...
                Finished with flag 0 after 80 iterations with relres 3.00475e-06 (from 1.39017e-05)
                Time needed 196.117 ms
                dx = 8.18141e-06, res_now = 3.00475e-06, res_old = 0.000127162
        Core 2
                Choosing iterative solver (local size 87552)...
                Finished with flag 0 after 80 iterations with relres 7.6319e-06 (from 1.14341e-05)
                Time needed 340.246 ms
                dx = 1.95852e-05, res_now = 7.6319e-06, res_old = 0.000154605
        Core 3
                Choosing iterative solver (local size 87040)...
                Finished with flag 0 after 80 iterations with relres 4.71959e-06 (from 1.41434e-05)
                Time needed 373.216 ms
                dx = 1.2303e-05, res_now = 4.71959e-06, res_old = 0.000124989
        Core 4
                Choosing iterative solver (local size 77824)...
                Finished with flag 0 after 80 iterations with relres 2.72318e-06 (from 1.74919e-05)
                Time needed 329.306 ms
                dx = 6.22547e-06, res_now = 2.72318e-06, res_old = 0.000101062
        Core 5
                Choosing iterative solver (local size 87552)...
                Finished with flag 0 after 80 iterations with relres 3.77222e-06 (from 2.28113e-05)
                Time needed 369.724 ms
                dx = 9.11199e-06, res_now = 3.77222e-06, res_old = 7.74952e-05
        Core 6
                Choosing iterative solver (local size 107008)...
                Finished with flag 0 after 80 iterations with relres 8.93457e-07 (from 0.00010787)
                Time needed 302.564 ms
                dx = 1.61717e-06, res_now = 8.93457e-07, res_old = 1.63879e-05
        Core 7
                Choosing iterative solver (local size 5376)...
                Finished with flag 0 after 80 iterations with relres 1.33219e-07 (from 0.000857222)
                Time needed 82.106 ms
                dx = 5.48779e-07, res_now = 1.33219e-07, res_old = 2.06221e-06
Solution rank [ 1 17 19 20 19 19 22 21 1 ]
Maxres 0.000154605
Time 2155.01 ms

Starting sweep 8
        Core 0
                Choosing iterative solver (local size 4352)...
                Finished with flag 0 after 80 iterations with relres 4.34749e-07 (from 0.000180152)
                Time needed 71.814 ms
                dx = 1.75874e-06, res_now = 4.34749e-07, res_old = 9.81267e-06
        Core 1
                Choosing iterative solver (local size 102144)...
                Finished with flag 0 after 80 iterations with relres 3.60253e-07 (from 0.000159874)
                Time needed 247.937 ms
                dx = 7.9549e-07, res_now = 3.60253e-07, res_old = 1.10573e-05
        Core 2
                Choosing iterative solver (local size 107520)...
                Finished with flag 0 after 80 iterations with relres 7.27459e-07 (from 9.66075e-05)
                Time needed 379.532 ms
                dx = 2.03312e-06, res_now = 7.27459e-07, res_old = 1.82984e-05
        Core 3
                Choosing iterative solver (local size 111872)...
                Finished with flag 0 after 80 iterations with relres 7.1912e-07 (from 0.000142913)
                Time needed 454.995 ms
                dx = 1.59779e-06, res_now = 7.1912e-07, res_old = 1.23695e-05
        Core 4
                Choosing iterative solver (local size 97280)...
                Finished with flag 0 after 80 iterations with relres 2.25273e-07 (from 0.000167124)
                Time needed 410.197 ms
                dx = 5.3144e-07, res_now = 2.25273e-07, res_old = 1.05776e-05
        Core 5
                Choosing iterative solver (local size 129536)...
                Finished with flag 0 after 80 iterations with relres 3.69624e-07 (from 0.000169438)
                Time needed 545.348 ms
                dx = 5.99467e-07, res_now = 3.69624e-07, res_old = 1.04331e-05
        Core 6
                Choosing iterative solver (local size 139776)...
                Finished with flag 0 after 80 iterations with relres 1.15083e-07 (from 0.00142775)
                Time needed 402.738 ms
                dx = 1.75731e-07, res_now = 1.15083e-07, res_old = 1.23815e-06
        Core 7
                Choosing iterative solver (local size 6400)...
                Finished with flag 0 after 80 iterations with relres 3.30769e-09 (from 0.0127578)
                Time needed 63.633 ms
                dx = 1.76064e-08, res_now = 3.30769e-09, res_old = 1.38564e-07
Solution rank [ 1 21 21 23 20 23 26 25 1 ]
Maxres 1.82984e-05
Time 2732.16 ms

Starting sweep 9
        Core 0
                Choosing iterative solver (local size 5376)...
                Finished with flag 0 after 80 iterations with relres 1.93496e-08 (from 0.00201292)
                Time needed 60.323 ms
                dx = 8.60513e-08, res_now = 1.93496e-08, res_old = 8.78209e-07
        Core 1
                Choosing iterative solver (local size 134400)...
                Finished with flag 0 after 80 iterations with relres 7.34548e-08 (from 0.0012428)
                Time needed 284.044 ms
                dx = 1.62034e-07, res_now = 7.34548e-08, res_old = 1.42241e-06
        Core 2
                Choosing iterative solver (local size 147200)...
                Finished with flag 0 after 80 iterations with relres 5.94483e-08 (from 0.000881885)
                Time needed 422.213 ms
                dx = 1.2289e-07, res_now = 5.94483e-08, res_old = 2.00453e-06
        Core 3
                Choosing iterative solver (local size 117760)...
                Finished with flag 0 after 80 iterations with relres 9.87087e-08 (from 0.00101082)
                Time needed 489.353 ms
                dx = 1.58656e-07, res_now = 9.87087e-08, res_old = 1.74884e-06
        Core 4
                Choosing iterative solver (local size 141312)...
                Finished with flag 0 after 80 iterations with relres 4.25944e-08 (from 0.00159416)
                Time needed 466.259 ms
                dx = 7.9085e-08, res_now = 4.25944e-08, res_old = 1.1089e-06
        Core 5
                Choosing iterative solver (local size 179712)...
                Finished with flag 0 after 80 iterations with relres 4.65398e-08 (from 0.00185744)
                Time needed 583.17 ms
                dx = 7.21484e-08, res_now = 4.65398e-08, res_old = 9.51724e-07
        Core 6
                Choosing iterative solver (local size 192000)...
                Finished with flag 0 after 80 iterations with relres 1.58085e-08 (from 0.00351722)
                Time needed 480.038 ms
                dx = 2.05083e-08, res_now = 1.58085e-08, res_old = 5.02604e-07
        Core 7
                Choosing iterative solver (local size 7424)...
                Finished with flag 1 after 80 iterations with relres 1.71225e-09 (from 0.111796)
                Time needed 66.872 ms
                dx = 3.54656e-09, res_now = 1.71225e-09, res_old = 1.58125e-08
Solution rank [ 1 25 25 23 24 27 30 29 1 ]
Maxres 2.00453e-06
Time 3016.91 ms

Starting sweep 10
        Core 0
                Choosing iterative solver (local size 6400)...
                Finished with flag 0 after 80 iterations with relres 2.92839e-09 (from 0.0124567)
                Time needed 60.341 ms
                dx = 6.1276e-09, res_now = 2.92839e-09, res_old = 1.41913e-07
        Core 1
                Choosing iterative solver (local size 134400)...
                Finished with flag 0 after 80 iterations with relres 1.56109e-08 (from 0.00820173)
                Time needed 279.687 ms
                dx = 2.92266e-08, res_now = 1.56109e-08, res_old = 2.15536e-07
        Core 2
                Choosing iterative solver (local size 170752)...
                Finished with flag 0 after 80 iterations with relres 8.75526e-09 (from 0.00817883)
                Time needed 475.361 ms
                dx = 1.78504e-08, res_now = 8.75526e-09, res_old = 2.16139e-07
        Core 3
                Choosing iterative solver (local size 153600)...
                Finished with flag 0 after 80 iterations with relres 1.74851e-08 (from 0.00509119)
                Time needed 488.842 ms
                dx = 2.61599e-08, res_now = 1.74851e-08, res_old = 3.47221e-07
        Core 4
                Choosing iterative solver (local size 179712)...
                Finished with flag 0 after 80 iterations with relres 4.87078e-09 (from 0.0118274)
                Time needed 571.268 ms
                dx = 9.68781e-09, res_now = 4.87078e-09, res_old = 1.49464e-07
        Core 5
                Choosing iterative solver (local size 199680)...
                Finished with flag 0 after 80 iterations with relres 1.12105e-08 (from 0.0118)
                Time needed 629.522 ms
                dx = 2.07325e-08, res_now = 1.12105e-08, res_old = 1.49811e-07
        Core 6
                Choosing iterative solver (local size 252416)...
                Finished with flag 0 after 80 iterations with relres 3.71388e-09 (from 0.0234078)
                Time needed 635.985 ms
                dx = 6.06579e-09, res_now = 3.71388e-09, res_old = 7.55205e-08
        Core 7
                Choosing iterative solver (local size 8448)...
                Finished with flag 1 after 6 iterations with relres 1.72194e-09 (from 0.476888)
                Time needed 6.157 ms
                dx = 5.15445e-11, res_now = 1.72194e-09, res_old = 3.70688e-09
Solution rank [ 1 21 29 25 26 26 34 33 1 ]
Maxres 3.47221e-07
Time 3368.51 ms

Starting sweep 11
        Core 0
                Choosing iterative solver (local size 5376)...
                Finished with flag 1 after 74 iterations with relres 1.73557e-09 (from 0.0819876)
                Time needed 54.274 ms
                dx = 4.1006e-09, res_now = 1.73557e-09, res_old = 2.15614e-08
        Core 1
                Choosing iterative solver (local size 155904)...
                Finished with flag 0 after 80 iterations with relres 2.49126e-09 (from 0.0302414)
                Time needed 375.396 ms
                dx = 5.24736e-09, res_now = 2.49126e-09, res_old = 5.84553e-08
        Core 2
                Choosing iterative solver (local size 160000)...
                Finished with flag 1 after 79 iterations with relres 1.74221e-09 (from 0.0347583)
                Time needed 482.488 ms
                dx = 3.42954e-09, res_now = 1.74221e-09, res_old = 5.08588e-08
        Core 3
                Choosing iterative solver (local size 173056)...
                Finished with flag 1 after 78 iterations with relres 1.75166e-09 (from 0.0344027)
                Time needed 564.239 ms
                dx = 3.04017e-09, res_now = 1.75166e-09, res_old = 5.13846e-08
        Core 4
                Choosing iterative solver (local size 166400)...
                Finished with flag 1 after 53 iterations with relres 1.74848e-09 (from 0.0742907)
                Time needed 360.13 ms
                dx = 7.23585e-10, res_now = 1.74848e-09, res_old = 2.37953e-08
        Core 5
                Choosing iterative solver (local size 217600)...
                Finished with flag 0 after 80 iterations with relres 3.79829e-09 (from 0.0460987)
                Time needed 668.546 ms
                dx = 7.51843e-09, res_now = 3.79829e-09, res_old = 3.83474e-08
        Core 6
                Choosing iterative solver (local size 321024)...
                Finished with flag 1 after 75 iterations with relres 1.76276e-09 (from 0.152624)
                Time needed 723.423 ms
                dx = 1.8139e-09, res_now = 1.76276e-09, res_old = 1.15825e-08
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 2 iterations with relres 1.70662e-09 (from 0.81524)
                Time needed 2.813 ms
                dx = 2.41031e-12, res_now = 1.70662e-09, res_old = 2.1684e-09
Solution rank [ 1 21 25 26 25 25 38 22 1 ]
Maxres 5.84553e-08
Time 3615.87 ms

Starting sweep 12
        Core 0
                Choosing iterative solver (local size 5376)...
                Finished with flag 1 after 4 iterations with relres 1.68538e-09 (from 0.289029)
                Time needed 3.847 ms
                dx = 8.9353e-12, res_now = 1.68538e-09, res_old = 6.11622e-09
        Core 1
                Choosing iterative solver (local size 140800)...
                Finished with flag 1 after 12 iterations with relres 1.74896e-09 (from 0.163235)
                Time needed 35.305 ms
                dx = 4.84312e-11, res_now = 1.74896e-09, res_old = 1.08296e-08
        Core 2
                Choosing iterative solver (local size 159744)...
                Finished with flag 1 after 8 iterations with relres 1.65256e-09 (from 0.25624)
                Time needed 44.347 ms
                dx = 2.61847e-11, res_now = 1.65256e-09, res_old = 6.89888e-09
        Core 3
                Choosing iterative solver (local size 153600)...
                Finished with flag 1 after 7 iterations with relres 1.73004e-09 (from 0.191273)
                Time needed 47.508 ms
                dx = 1.25729e-11, res_now = 1.73004e-09, res_old = 9.2421e-09
        Core 4
                Choosing iterative solver (local size 153600)...
                Finished with flag 1 after 9 iterations with relres 1.74416e-09 (from 0.120395)
                Time needed 56.885 ms
                dx = 2.61427e-11, res_now = 1.74416e-09, res_old = 1.4683e-08
        Core 5
                Choosing iterative solver (local size 233472)...
                Finished with flag 1 after 71 iterations with relres 1.74997e-09 (from 0.153176)
                Time needed 626.417 ms
                dx = 1.62691e-09, res_now = 1.74997e-09, res_old = 1.15407e-08
        Core 6
                Choosing iterative solver (local size 129536)...
                Finished with flag 1 after 9 iterations with relres 1.75027e-09 (from 0.516988)
                Time needed 38.441 ms
                dx = 7.35425e-11, res_now = 1.75027e-09, res_old = 3.41935e-09
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.72528e-09 (from 0.818898)
                Time needed 2.034 ms
                dx = 7.92613e-13, res_now = 1.72528e-09, res_old = 2.15872e-09
Solution rank [ 1 22 24 24 24 24 23 22 1 ]
Maxres 1.4683e-08
Time 1215.58 ms

Starting sweep 13
        Core 0
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 2 iterations with relres 1.5521e-09 (from 0.401053)
                Time needed 2.228 ms
                dx = 2.77285e-12, res_now = 1.5521e-09, res_old = 4.40781e-09
        Core 1
                Choosing iterative solver (local size 135168)...
                Finished with flag 1 after 5 iterations with relres 1.71606e-09 (from 0.252144)
                Time needed 17.001 ms
                dx = 8.10491e-12, res_now = 1.71606e-09, res_old = 7.01094e-09
        Core 2
                Choosing iterative solver (local size 147456)...
                Finished with flag 1 after 5 iterations with relres 1.75041e-09 (from 0.301712)
                Time needed 27.209 ms
                dx = 8.51595e-12, res_now = 1.75041e-09, res_old = 5.85913e-09
        Core 3
                Choosing iterative solver (local size 147456)...
                Finished with flag 1 after 6 iterations with relres 1.69269e-09 (from 0.269019)
                Time needed 39.813 ms
                dx = 1.05837e-11, res_now = 1.69269e-09, res_old = 6.57116e-09
        Core 4
                Choosing iterative solver (local size 147456)...
                Finished with flag 1 after 6 iterations with relres 1.75541e-09 (from 0.234904)
                Time needed 37.964 ms
                dx = 1.17993e-11, res_now = 1.75541e-09, res_old = 7.5255e-09
        Core 5
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 10 iterations with relres 1.71809e-09 (from 0.234484)
                Time needed 53.664 ms
                dx = 2.69435e-11, res_now = 1.71809e-09, res_old = 7.53897e-09
        Core 6
                Choosing iterative solver (local size 129536)...
                Finished with flag 1 after 6 iterations with relres 1.7452e-09 (from 0.552582)
                Time needed 26.699 ms
                dx = 2.78372e-11, res_now = 1.7452e-09, res_old = 3.19911e-09
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.69481e-09 (from 0.835815)
                Time needed 2.117 ms
                dx = 7.85563e-13, res_now = 1.69481e-09, res_old = 2.11502e-09
Solution rank [ 1 22 24 24 24 23 23 22 1 ]
Maxres 7.53897e-09
Time 432.161 ms

Starting sweep 14 (last one)
        Core 0
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 2 iterations with relres 1.36106e-09 (from 0.490868)
                Time needed 3.351 ms
                dx = 3.06325e-12, res_now = 1.36106e-09, res_old = 3.60131e-09
        Core 1
                Choosing iterative solver (local size 110592)...
                Finished with flag 1 after 3 iterations with relres 1.56994e-09 (from 0.331585)
                Time needed 12.705 ms
                dx = 3.66024e-12, res_now = 1.56994e-09, res_old = 5.33127e-09
        Core 2
                Choosing iterative solver (local size 122880)...
                Finished with flag 1 after 2 iterations with relres 1.60155e-09 (from 0.381848)
                Time needed 19.846 ms
                dx = 2.59678e-12, res_now = 1.60155e-09, res_old = 4.62951e-09
        Core 3
                Choosing iterative solver (local size 122880)...
                Finished with flag 1 after 3 iterations with relres 1.51754e-09 (from 0.302895)
                Time needed 29.484 ms
                dx = 3.66286e-12, res_now = 1.51754e-09, res_old = 5.83623e-09
        Core 4
                Choosing iterative solver (local size 117760)...
                Finished with flag 1 after 3 iterations with relres 1.47395e-09 (from 0.379776)
                Time needed 28.75 ms
                dx = 4.03126e-12, res_now = 1.47395e-09, res_old = 4.65476e-09
        Core 5
                Choosing iterative solver (local size 111872)...
                Finished with flag 1 after 3 iterations with relres 1.67522e-09 (from 0.398863)
                Time needed 26.091 ms
                dx = 4.06173e-12, res_now = 1.67522e-09, res_old = 4.43201e-09
        Core 6
                Choosing iterative solver (local size 107008)...
                Finished with flag 1 after 1 iterations with relres 1.56962e-09 (from 1.0972)
                Time needed 11.176 ms
                dx = 6.39495e-13, res_now = 1.56962e-09, res_old = 1.61115e-09
        Core 7
                Choosing iterative solver (local size 4608)...
                Finished with flag 1 after 1 iterations with relres 2.43739e-10 (from 6.26761)
                Time needed 1.791 ms
                dx = 2.04451e-13, res_now = 2.43739e-10, res_old = 2.82048e-10
Solution rank [ 1 18 20 20 20 19 19 18 1 ]
Maxres 5.83623e-09
Time 299.792 ms

Finished after 14 sweeps and 22.6956 seconds

Relative residual:  tensor(4.9297e-09, dtype=torch.float64)
Solver time:  0:00:22.697280

Display the structure of the TT

[9]:
print(x)
TT with sizes and ranks:
N = [256, 256, 256, 256, 256, 256, 256, 256]
R = [1, 18, 20, 20, 20, 19, 19, 18, 1]

Device: cpu, dtype: torch.float64
#entries 583424 compression 3.162747841400915e-14

Try one more time on the GPU (if available).

[11]:
if tn.cuda.is_available():
    cuda_dev = 'cuda:0'
    time = datetime.datetime.now()
    x = tntt.solvers.amen_solve(L_tt.to(cuda_dev), b_tt.to(cuda_dev) ,x0 = b_tt.to(cuda_dev), nswp = 20, eps = 1e-8, verbose = True, preconditioner=None)
    time = datetime.datetime.now() - time
    x = x.cpu()
    print('Relative residual: ',(L_tt@x-b_tt).norm()/b_tt.norm())
    print('Solver time: ',time)
else:
    print('GPU not available...')
Starting AMEn solve with:
        epsilon              : 1e-08
        sweeps               : 20
        local iterations     : 40
        resets               : 2
        local preconditioner : N


Starting sweep 1
        Core 0
                Choosing direct solver (local size 256)...
                dx = 1.00022, res_now = 3.80241e-15, res_old = 1.00025
        Core 1
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 3.03976e-07 (from 1.12188e-09)
                Time needed 50.592 ms
                dx = 0.162228, res_now = 3.03976e-07, res_old = 1.57572
        Core 2
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 5.9404e-07 (from 9.82965e-10)
                Time needed 56.085 ms
                dx = 0.178782, res_now = 5.9404e-07, res_old = 1.7984
        Core 3
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 1.95171e-06 (from 8.42061e-10)
                Time needed 50.715 ms
                dx = 0.201372, res_now = 1.95171e-06, res_old = 2.09933
        Core 4
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 1.13927e-05 (from 6.99929e-10)
                Time needed 54.743 ms
                dx = 0.234093, res_now = 1.13927e-05, res_old = 2.52564
        Core 5
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 5.68151e-05 (from 5.56343e-10)
                Time needed 49.925 ms
                dx = 0.285461, res_now = 5.68151e-05, res_old = 3.17748
        Core 6
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.000383669 (from 4.10711e-10)
                Time needed 50.479 ms
                dx = 0.376859, res_now = 0.000383669, res_old = 4.30417
        Core 7
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.0104981 (from 2.63017e-10)
                Time needed 49.494 ms
                dx = 0.578521, res_now = 0.0104981, res_old = 6.72112
Solution rank [ 1 5 5 5 5 5 5 5 1 ]
Maxres 6.72112
Time 396.474 ms

Starting sweep 2
        Core 0
                Choosing iterative solver (local size 1280)...
                Finished with flag 0 after 80 iterations with relres 0.015867 (from 1.51816e-09)
                Time needed 50.06 ms
                dx = 0.167332, res_now = 0.015867, res_old = 1.16442
        Core 1
                Choosing iterative solver (local size 7680)...
                Finished with flag 0 after 80 iterations with relres 0.031902 (from 1.17296e-09)
                Time needed 59.738 ms
                dx = 0.178384, res_now = 0.031902, res_old = 1.5071
        Core 2
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0500298 (from 6.84387e-10)
                Time needed 66.412 ms
                dx = 0.160649, res_now = 0.0500298, res_old = 2.58299
        Core 3
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0633123 (from 6.75131e-10)
                Time needed 72.265 ms
                dx = 0.163413, res_now = 0.0633123, res_old = 2.61841
        Core 4
                Choosing iterative solver (local size 7680)...
                Finished with flag 0 after 80 iterations with relres 0.048629 (from 6.93307e-10)
                Time needed 72.327 ms
                dx = 0.180823, res_now = 0.048629, res_old = 2.54976
        Core 5
                Choosing iterative solver (local size 7680)...
                Finished with flag 0 after 80 iterations with relres 0.0511481 (from 7.49026e-10)
                Time needed 72.961 ms
                dx = 0.162795, res_now = 0.0511481, res_old = 2.36009
        Core 6
                Choosing iterative solver (local size 11520)...
                Finished with flag 0 after 80 iterations with relres 0.0626577 (from 9.98891e-10)
                Time needed 71.972 ms
                dx = 0.134086, res_now = 0.0626577, res_old = 1.76973
        Core 7
                Choosing iterative solver (local size 2304)...
                Finished with flag 0 after 80 iterations with relres 0.0380041 (from 2.6048e-09)
                Time needed 50.236 ms
                dx = 0.0816631, res_now = 0.0380041, res_old = 0.678659
Solution rank [ 1 6 9 9 6 6 9 9 1 ]
Maxres 2.61841
Time 571.607 ms

Starting sweep 3
        Core 0
                Choosing iterative solver (local size 1536)...
                Finished with flag 0 after 80 iterations with relres 0.0418839 (from 1.59849e-09)
                Time needed 50.264 ms
                dx = 0.0682368, res_now = 0.0418839, res_old = 1.1059
        Core 1
                Choosing iterative solver (local size 23040)...
                Finished with flag 0 after 80 iterations with relres 0.0331869 (from 1.74853e-09)
                Time needed 62.077 ms
                dx = 0.0551896, res_now = 0.0331869, res_old = 1.011
        Core 2
                Choosing iterative solver (local size 18432)...
                Finished with flag 0 after 80 iterations with relres 0.0238549 (from 3.92712e-09)
                Time needed 83.902 ms
                dx = 0.0613911, res_now = 0.0238549, res_old = 0.450143
        Core 3
                Choosing iterative solver (local size 19968)...
                Finished with flag 0 after 80 iterations with relres 0.0261808 (from 2.53909e-09)
                Time needed 95.81 ms
                dx = 0.04825, res_now = 0.0261808, res_old = 0.696221
        Core 4
                Choosing iterative solver (local size 15360)...
                Finished with flag 0 after 80 iterations with relres 0.0231217 (from 4.51109e-09)
                Time needed 73.142 ms
                dx = 0.0315142, res_now = 0.0231217, res_old = 0.391872
        Core 5
                Choosing iterative solver (local size 18432)...
                Finished with flag 0 after 80 iterations with relres 0.0143334 (from 5.31685e-09)
                Time needed 94.972 ms
                dx = 0.0397164, res_now = 0.0143334, res_old = 0.332484
        Core 6
                Choosing iterative solver (local size 29952)...
                Finished with flag 0 after 80 iterations with relres 0.0172939 (from 5.93701e-09)
                Time needed 96.303 ms
                dx = 0.0371247, res_now = 0.0172939, res_old = 0.297754
        Core 7
                Choosing iterative solver (local size 3328)...
                Finished with flag 0 after 80 iterations with relres 0.0105622 (from 1.0612e-08)
                Time needed 50.343 ms
                dx = 0.0244473, res_now = 0.0105622, res_old = 0.166582
Solution rank [ 1 10 8 13 10 8 13 13 1 ]
Maxres 1.1059
Time 666.818 ms

Starting sweep 4
        Core 0
                Choosing iterative solver (local size 2560)...
                Finished with flag 0 after 80 iterations with relres 0.00673767 (from 7.5402e-09)
                Time needed 50.788 ms
                dx = 0.0263428, res_now = 0.00673767, res_old = 0.234446
        Core 1
                Choosing iterative solver (local size 20480)...
                Finished with flag 0 after 80 iterations with relres 0.00836376 (from 1.19508e-08)
                Time needed 61.967 ms
                dx = 0.0170012, res_now = 0.00836376, res_old = 0.14792
        Core 2
                Choosing iterative solver (local size 36608)...
                Finished with flag 0 after 80 iterations with relres 0.00497299 (from 1.58634e-08)
                Time needed 84.714 ms
                dx = 0.012072, res_now = 0.00497299, res_old = 0.111437
        Core 3
                Choosing iterative solver (local size 43520)...
                Finished with flag 0 after 80 iterations with relres 0.00660013 (from 1.47626e-08)
                Time needed 103.66 ms
                dx = 0.0115263, res_now = 0.00660013, res_old = 0.119747
        Core 4
                Choosing iterative solver (local size 22528)...
                Finished with flag 0 after 80 iterations with relres 0.00460054 (from 1.61525e-08)
                Time needed 95.789 ms
                dx = 0.0118596, res_now = 0.00460054, res_old = 0.109442
        Core 5
                Choosing iterative solver (local size 36608)...
                Finished with flag 0 after 80 iterations with relres 0.00212978 (from 1.89268e-08)
                Time needed 95.175 ms
                dx = 0.00526326, res_now = 0.00212978, res_old = 0.0934002
        Core 6
                Choosing iterative solver (local size 36608)...
                Finished with flag 0 after 80 iterations with relres 0.00239492 (from 2.41421e-08)
                Time needed 98.187 ms
                dx = 0.00637493, res_now = 0.00239492, res_old = 0.0732235
        Core 7
                Choosing iterative solver (local size 2816)...
                Finished with flag 0 after 80 iterations with relres 0.000638102 (from 1.44174e-07)
                Time needed 50.583 ms
                dx = 0.00270128, res_now = 0.000638102, res_old = 0.0122613
Solution rank [ 1 10 11 17 11 11 11 11 1 ]
Maxres 0.234446
Time 714.9 ms

Starting sweep 5
        Core 0
                Choosing iterative solver (local size 2560)...
                Finished with flag 0 after 80 iterations with relres 0.000699438 (from 6.1891e-08)
                Time needed 50.698 ms
                dx = 0.00289298, res_now = 0.000699438, res_old = 0.0285626
        Core 1
                Choosing iterative solver (local size 33792)...
                Finished with flag 0 after 80 iterations with relres 0.0009356 (from 7.56992e-08)
                Time needed 74.918 ms
                dx = 0.00169719, res_now = 0.0009356, res_old = 0.0233525
        Core 2
                Choosing iterative solver (local size 52224)...
                Finished with flag 0 after 80 iterations with relres 0.00050212 (from 6.01067e-08)
                Time needed 121.858 ms
                dx = 0.00131698, res_now = 0.00050212, res_old = 0.0294105
        Core 3
                Choosing iterative solver (local size 59136)...
                Finished with flag 0 after 80 iterations with relres 0.000750441 (from 6.41931e-08)
                Time needed 146.091 ms
                dx = 0.00190719, res_now = 0.000750441, res_old = 0.0275383
        Core 4
                Choosing iterative solver (local size 36608)...
                Finished with flag 0 after 80 iterations with relres 0.000317133 (from 1.39246e-07)
                Time needed 94.735 ms
                dx = 0.000986581, res_now = 0.000317133, res_old = 0.0126953
        Core 5
                Choosing iterative solver (local size 42240)...
                Finished with flag 0 after 80 iterations with relres 0.000159019 (from 3.5446e-07)
                Time needed 102.171 ms
                dx = 0.000321059, res_now = 0.000159019, res_old = 0.00498721
        Core 6
                Choosing iterative solver (local size 42240)...
                Finished with flag 0 after 80 iterations with relres 0.000281302 (from 2.38104e-07)
                Time needed 142.334 ms
                dx = 0.000590865, res_now = 0.000281302, res_old = 0.00742435
        Core 7
                Choosing iterative solver (local size 3840)...
                Finished with flag 0 after 80 iterations with relres 5.48164e-05 (from 3.14187e-06)
                Time needed 50.469 ms
                dx = 9.48092e-05, res_now = 5.48164e-05, res_old = 0.000562648
Solution rank [ 1 12 12 21 13 15 15 15 1 ]
Maxres 0.0294105
Time 855.034 ms

Starting sweep 6
        Core 0
                Choosing iterative solver (local size 3072)...
                Finished with flag 0 after 80 iterations with relres 6.25343e-05 (from 8.29497e-07)
                Time needed 52.66 ms
                dx = 0.000296777, res_now = 6.25343e-05, res_old = 0.00213113
        Core 1
                Choosing iterative solver (local size 46080)...
                Finished with flag 0 after 80 iterations with relres 0.000106668 (from 6.67088e-07)
                Time needed 101.014 ms
                dx = 0.000288519, res_now = 0.000106668, res_old = 0.00264997
        Core 2
                Choosing iterative solver (local size 86016)...
                Finished with flag 0 after 80 iterations with relres 9.61793e-05 (from 7.17708e-07)
                Time needed 158.202 ms
                dx = 0.000218815, res_now = 9.61793e-05, res_old = 0.00246307
        Core 3
                Choosing iterative solver (local size 56576)...
                Finished with flag 0 after 80 iterations with relres 4.83301e-05 (from 9.51403e-07)
                Time needed 146.615 ms
                dx = 0.000108769, res_now = 4.83301e-05, res_old = 0.00185806
        Core 4
                Choosing iterative solver (local size 65280)...
                Finished with flag 0 after 80 iterations with relres 4.12296e-05 (from 1.9095e-06)
                Time needed 148.326 ms
                dx = 5.92612e-05, res_now = 4.12296e-05, res_old = 0.000925776
        Core 5
                Choosing iterative solver (local size 65280)...
                Finished with flag 0 after 80 iterations with relres 3.00437e-05 (from 1.64676e-06)
                Time needed 148.734 ms
                dx = 7.19659e-05, res_now = 3.00437e-05, res_old = 0.00107348
        Core 6
                Choosing iterative solver (local size 72960)...
                Finished with flag 0 after 80 iterations with relres 3.54605e-05 (from 3.18765e-06)
                Time needed 147.546 ms
                dx = 8.34086e-05, res_now = 3.54605e-05, res_old = 0.000554568
        Core 7
                Choosing iterative solver (local size 4352)...
                Finished with flag 0 after 80 iterations with relres 3.27163e-06 (from 3.97373e-05)
                Time needed 51.393 ms
                dx = 1.35342e-05, res_now = 3.27163e-06, res_old = 4.44863e-05
Solution rank [ 1 15 16 17 17 17 19 17 1 ]
Maxres 0.00264997
Time 1052.64 ms

Starting sweep 7
        Core 0
                Choosing iterative solver (local size 3840)...
                Finished with flag 0 after 80 iterations with relres 8.21288e-06 (from 9.00825e-06)
                Time needed 50.869 ms
                dx = 3.5975e-05, res_now = 8.21288e-06, res_old = 0.000196239
        Core 1
                Choosing iterative solver (local size 77824)...
                Finished with flag 0 after 80 iterations with relres 1.20709e-05 (from 6.10482e-06)
                Time needed 101.164 ms
                dx = 2.84578e-05, res_now = 1.20709e-05, res_old = 0.000289569
        Core 2
                Choosing iterative solver (local size 87040)...
                Finished with flag 0 after 80 iterations with relres 1.40472e-05 (from 6.52769e-06)
                Time needed 160.074 ms
                dx = 3.63059e-05, res_now = 1.40472e-05, res_old = 0.00027081
        Core 3
                Choosing iterative solver (local size 82688)...
                Finished with flag 0 after 80 iterations with relres 4.1683e-06 (from 8.12212e-06)
                Time needed 191.457 ms
                dx = 8.79162e-06, res_now = 4.1683e-06, res_old = 0.000217648
        Core 4
                Choosing iterative solver (local size 91392)...
                Finished with flag 0 after 80 iterations with relres 6.82361e-06 (from 1.24031e-05)
                Time needed 194.622 ms
                dx = 1.08804e-05, res_now = 6.82361e-06, res_old = 0.000142526
        Core 5
                Choosing iterative solver (local size 102144)...
                Finished with flag 0 after 80 iterations with relres 7.60589e-06 (from 1.46352e-05)
                Time needed 196.999 ms
                dx = 1.05553e-05, res_now = 7.60589e-06, res_old = 0.000120788
        Core 6
                Choosing iterative solver (local size 87040)...
                Finished with flag 0 after 80 iterations with relres 4.31245e-06 (from 1.21131e-05)
                Time needed 148.624 ms
                dx = 6.72892e-06, res_now = 4.31245e-06, res_old = 0.000145939
        Core 7
                Choosing iterative solver (local size 5376)...
                Finished with flag 0 after 80 iterations with relres 4.06713e-07 (from 0.000292801)
                Time needed 51.546 ms
                dx = 1.53479e-06, res_now = 4.06713e-07, res_old = 6.03742e-06
Solution rank [ 1 19 20 19 21 21 20 21 1 ]
Maxres 0.000289569
Time 1196.01 ms

Starting sweep 8
        Core 0
                Choosing iterative solver (local size 4864)...
                Finished with flag 0 after 80 iterations with relres 8.36645e-07 (from 6.19075e-05)
                Time needed 55.65 ms
                dx = 2.29974e-06, res_now = 8.36645e-07, res_old = 2.8555e-05
        Core 1
                Choosing iterative solver (local size 117760)...
                Finished with flag 0 after 80 iterations with relres 2.25188e-06 (from 3.71368e-05)
                Time needed 128.37 ms
                dx = 4.74249e-06, res_now = 2.25188e-06, res_old = 4.76015e-05
        Core 2
                Choosing iterative solver (local size 102144)...
                Finished with flag 0 after 80 iterations with relres 1.75598e-06 (from 4.81325e-05)
                Time needed 160.84 ms
                dx = 2.39278e-06, res_now = 1.75598e-06, res_old = 3.67271e-05
        Core 3
                Choosing iterative solver (local size 123648)...
                Finished with flag 0 after 80 iterations with relres 1.0198e-06 (from 9.15678e-05)
                Time needed 243.534 ms
                dx = 1.92886e-06, res_now = 1.0198e-06, res_old = 1.93055e-05
        Core 4
                Choosing iterative solver (local size 112896)...
                Finished with flag 0 after 80 iterations with relres 1.14973e-06 (from 7.65133e-05)
                Time needed 199.709 ms
                dx = 2.21528e-06, res_now = 1.14973e-06, res_old = 2.3104e-05
        Core 5
                Choosing iterative solver (local size 128000)...
                Finished with flag 0 after 80 iterations with relres 1.4691e-06 (from 6.80224e-05)
                Time needed 243.762 ms
                dx = 2.87973e-06, res_now = 1.4691e-06, res_old = 2.5988e-05
        Core 6
                Choosing iterative solver (local size 129024)...
                Finished with flag 0 after 80 iterations with relres 6.03489e-07 (from 9.97236e-05)
                Time needed 196.714 ms
                dx = 9.91745e-07, res_now = 6.03489e-07, res_old = 1.77267e-05
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 0 after 80 iterations with relres 3.93528e-08 (from 0.00257389)
                Time needed 52.107 ms
                dx = 2.56513e-07, res_now = 3.93528e-08, res_old = 6.86807e-07
Solution rank [ 1 23 21 23 21 25 24 22 1 ]
Maxres 4.76015e-05
Time 1403.2 ms

Starting sweep 9
        Core 0
                Choosing iterative solver (local size 5888)...
                Finished with flag 0 after 80 iterations with relres 1.19978e-07 (from 0.000566685)
                Time needed 55.835 ms
                dx = 4.82923e-07, res_now = 1.19978e-07, res_old = 3.11949e-06
        Core 1
                Choosing iterative solver (local size 145152)...
                Finished with flag 0 after 80 iterations with relres 2.3584e-07 (from 0.000341401)
                Time needed 126.97 ms
                dx = 5.76025e-07, res_now = 2.3584e-07, res_old = 5.17798e-06
        Core 2
                Choosing iterative solver (local size 135424)...
                Finished with flag 0 after 80 iterations with relres 1.75276e-07 (from 0.000382275)
                Time needed 199.546 ms
                dx = 2.70994e-07, res_now = 1.75276e-07, res_old = 4.62433e-06
        Core 3
                Choosing iterative solver (local size 123648)...
                Finished with flag 0 after 80 iterations with relres 1.71806e-07 (from 0.000650124)
                Time needed 243.396 ms
                dx = 3.67211e-07, res_now = 1.71806e-07, res_old = 2.71912e-06
        Core 4
                Choosing iterative solver (local size 160000)...
                Finished with flag 0 after 80 iterations with relres 2.29502e-07 (from 0.000466189)
                Time needed 290.595 ms
                dx = 3.06747e-07, res_now = 2.29502e-07, res_old = 3.79195e-06
        Core 5
                Choosing iterative solver (local size 178176)...
                Finished with flag 0 after 80 iterations with relres 2.11133e-07 (from 0.000356779)
                Time needed 420.968 ms
                dx = 3.858e-07, res_now = 2.11133e-07, res_old = 4.9548e-06
        Core 6
                Choosing iterative solver (local size 157696)...
                Finished with flag 0 after 80 iterations with relres 8.62579e-08 (from 0.000926516)
                Time needed 241.73 ms
                dx = 1.43002e-07, res_now = 8.62579e-08, res_old = 1.90797e-06
        Core 7
                Choosing iterative solver (local size 6656)...
                Finished with flag 0 after 80 iterations with relres 9.4085e-09 (from 0.0201259)
                Time needed 51.949 ms
                dx = 2.93499e-08, res_now = 9.4085e-09, res_old = 8.78355e-08
Solution rank [ 1 27 23 23 25 29 28 26 1 ]
Maxres 5.17798e-06
Time 1758.28 ms

Starting sweep 10
        Core 0
                Choosing iterative solver (local size 6912)...
                Finished with flag 0 after 80 iterations with relres 1.72384e-08 (from 0.0046336)
                Time needed 55.927 ms
                dx = 5.27699e-08, res_now = 1.72384e-08, res_old = 3.81511e-07
        Core 1
                Choosing iterative solver (local size 182528)...
                Finished with flag 0 after 80 iterations with relres 2.73321e-08 (from 0.00287463)
                Time needed 150.521 ms
                dx = 5.31336e-08, res_now = 2.73321e-08, res_old = 6.14955e-07
        Core 2
                Choosing iterative solver (local size 158976)...
                Finished with flag 0 after 80 iterations with relres 2.63825e-08 (from 0.00372511)
                Time needed 230.62 ms
                dx = 4.48571e-08, res_now = 2.63825e-08, res_old = 4.74554e-07
        Core 3
                Choosing iterative solver (local size 160000)...
                Finished with flag 0 after 80 iterations with relres 1.82521e-08 (from 0.00367171)
                Time needed 290.461 ms
                dx = 3.11738e-08, res_now = 1.82521e-08, res_old = 4.81456e-07
        Core 4
                Choosing iterative solver (local size 215296)...
                Finished with flag 0 after 80 iterations with relres 3.00877e-08 (from 0.00226903)
                Time needed 338.031 ms
                dx = 5.36355e-08, res_now = 3.00877e-08, res_old = 7.79086e-07
        Core 5
                Choosing iterative solver (local size 193536)...
                Finished with flag 0 after 80 iterations with relres 1.49718e-08 (from 0.00358503)
                Time needed 422.516 ms
                dx = 2.78158e-08, res_now = 1.49718e-08, res_old = 4.93097e-07
        Core 6
                Choosing iterative solver (local size 179712)...
                Finished with flag 0 after 80 iterations with relres 8.15032e-09 (from 0.007371)
                Time needed 243.167 ms
                dx = 1.6803e-08, res_now = 8.15032e-09, res_old = 2.39827e-07
        Core 7
                Choosing iterative solver (local size 7680)...
                Finished with flag 1 after 68 iterations with relres 1.73718e-09 (from 0.209899)
                Time needed 39.795 ms
                dx = 2.00851e-09, res_now = 1.73718e-09, res_old = 8.42197e-09
Solution rank [ 1 31 27 25 29 27 27 30 1 ]
Maxres 7.79086e-07
Time 1960.88 ms

Starting sweep 11
        Core 0
                Choosing iterative solver (local size 7936)...
                Finished with flag 1 after 74 iterations with relres 1.75467e-09 (from 0.0236432)
                Time needed 49.129 ms
                dx = 4.53517e-09, res_now = 1.75467e-09, res_old = 7.47686e-08
        Core 1
                Choosing iterative solver (local size 158976)...
                Finished with flag 0 after 80 iterations with relres 3.98157e-09 (from 0.0150717)
                Time needed 151.234 ms
                dx = 8.3438e-09, res_now = 3.98157e-09, res_old = 1.17291e-07
        Core 2
                Choosing iterative solver (local size 166400)...
                Finished with flag 0 after 80 iterations with relres 3.44523e-09 (from 0.0156798)
                Time needed 232.104 ms
                dx = 6.05687e-09, res_now = 3.44523e-09, res_old = 1.12742e-07
        Core 3
                Choosing iterative solver (local size 185600)...
                Finished with flag 0 after 80 iterations with relres 3.48653e-09 (from 0.035096)
                Time needed 426.313 ms
                dx = 6.79284e-09, res_now = 3.48653e-09, res_old = 5.03695e-08
        Core 4
                Choosing iterative solver (local size 207360)...
                Finished with flag 0 after 80 iterations with relres 3.0563e-09 (from 0.0387987)
                Time needed 321.109 ms
                dx = 5.25471e-09, res_now = 3.0563e-09, res_old = 4.55625e-08
        Core 5
                Choosing iterative solver (local size 186624)...
                Finished with flag 1 after 61 iterations with relres 1.74632e-09 (from 0.0437416)
                Time needed 322.051 ms
                dx = 1.28997e-09, res_now = 1.74632e-09, res_old = 4.04139e-08
        Core 6
                Choosing iterative solver (local size 184320)...
                Finished with flag 1 after 72 iterations with relres 1.71683e-09 (from 0.0702795)
                Time needed 217.118 ms
                dx = 1.85506e-09, res_now = 1.71683e-09, res_old = 2.51534e-08
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.68707e-09 (from 0.899157)
                Time needed 0.659 ms
                dx = 9.67935e-13, res_now = 1.68707e-09, res_old = 1.96603e-09
Solution rank [ 1 23 26 25 30 27 24 22 1 ]
Maxres 1.17291e-07
Time 1975.16 ms

Starting sweep 12
        Core 0
                Choosing iterative solver (local size 5888)...
                Finished with flag 1 after 19 iterations with relres 1.76053e-09 (from 0.254848)
                Time needed 8.709 ms
                dx = 2.37954e-10, res_now = 1.76053e-09, res_old = 6.93656e-09
        Core 1
                Choosing iterative solver (local size 146432)...
                Finished with flag 1 after 39 iterations with relres 1.75273e-09 (from 0.111683)
                Time needed 63.539 ms
                dx = 6.10997e-10, res_now = 1.75273e-09, res_old = 1.58284e-08
        Core 2
                Choosing iterative solver (local size 160000)...
                Finished with flag 1 after 22 iterations with relres 1.7621e-09 (from 0.148265)
                Time needed 65.28 ms
                dx = 2.17345e-10, res_now = 1.7621e-09, res_old = 1.1923e-08
        Core 3
                Choosing iterative solver (local size 184320)...
                Finished with flag 1 after 60 iterations with relres 1.76222e-09 (from 0.149988)
                Time needed 320.443 ms
                dx = 1.23221e-09, res_now = 1.76222e-09, res_old = 1.1786e-08
        Core 4
                Choosing iterative solver (local size 158976)...
                Finished with flag 1 after 46 iterations with relres 1.76237e-09 (from 0.1896)
                Time needed 173.71 ms
                dx = 6.40444e-10, res_now = 1.76237e-09, res_old = 9.32365e-09
        Core 5
                Choosing iterative solver (local size 141312)...
                Finished with flag 1 after 14 iterations with relres 1.69016e-09 (from 0.417831)
                Time needed 45.238 ms
                dx = 9.57588e-11, res_now = 1.69016e-09, res_old = 4.23082e-09
        Core 6
                Choosing iterative solver (local size 129536)...
                Finished with flag 1 after 7 iterations with relres 1.66911e-09 (from 0.475457)
                Time needed 20.959 ms
                dx = 2.38051e-11, res_now = 1.66911e-09, res_old = 3.71804e-09
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.65367e-09 (from 0.74809)
                Time needed 0.631 ms
                dx = 8.87253e-13, res_now = 1.65367e-09, res_old = 2.36304e-09
Solution rank [ 1 22 25 24 23 23 23 22 1 ]
Maxres 1.58284e-08
Time 936.523 ms

Starting sweep 13
        Core 0
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 2 iterations with relres 1.28617e-09 (from 0.251033)
                Time needed 1.093 ms
                dx = 3.4751e-12, res_now = 1.28617e-09, res_old = 7.04196e-09
        Core 1
                Choosing iterative solver (local size 140800)...
                Finished with flag 1 after 6 iterations with relres 1.71449e-09 (from 0.142137)
                Time needed 10.636 ms
                dx = 1.03098e-11, res_now = 1.71449e-09, res_old = 1.24371e-08
        Core 2
                Choosing iterative solver (local size 147456)...
                Finished with flag 1 after 9 iterations with relres 1.72536e-09 (from 0.152134)
                Time needed 24.977 ms
                dx = 2.36773e-11, res_now = 1.72536e-09, res_old = 1.16198e-08
        Core 3
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 15 iterations with relres 1.72289e-09 (from 0.100777)
                Time needed 47.764 ms
                dx = 6.39156e-11, res_now = 1.72289e-09, res_old = 1.75414e-08
        Core 4
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 11 iterations with relres 1.72084e-09 (from 0.226403)
                Time needed 37.509 ms
                dx = 4.19339e-11, res_now = 1.72084e-09, res_old = 7.80806e-09
        Core 5
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 4 iterations with relres 1.66502e-09 (from 0.438618)
                Time needed 19.115 ms
                dx = 5.4306e-12, res_now = 1.66502e-09, res_old = 4.03031e-09
        Core 6
                Choosing iterative solver (local size 129536)...
                Finished with flag 1 after 5 iterations with relres 1.63799e-09 (from 0.5512)
                Time needed 16.976 ms
                dx = 1.62868e-11, res_now = 1.63799e-09, res_old = 3.20713e-09
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.55732e-09 (from 0.805674)
                Time needed 0.653 ms
                dx = 7.9711e-13, res_now = 1.55732e-09, res_old = 2.19415e-09
Solution rank [ 1 22 24 23 23 23 23 22 1 ]
Maxres 1.75414e-08
Time 320.59 ms

Starting sweep 14
        Core 0
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.26335e-09 (from 0.578092)
                Time needed 0.905 ms
                dx = 1.38103e-12, res_now = 1.26335e-09, res_old = 3.05794e-09
        Core 1
                Choosing iterative solver (local size 135168)...
                Finished with flag 1 after 3 iterations with relres 1.63465e-09 (from 0.293481)
                Time needed 7.237 ms
                dx = 3.90547e-12, res_now = 1.63465e-09, res_old = 6.02345e-09
        Core 2
                Choosing iterative solver (local size 141312)...
                Finished with flag 1 after 5 iterations with relres 1.64806e-09 (from 0.224868)
                Time needed 16.95 ms
                dx = 9.74615e-12, res_now = 1.64806e-09, res_old = 7.86136e-09
        Core 3
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 7 iterations with relres 1.66827e-09 (from 0.281059)
                Time needed 26.666 ms
                dx = 1.52581e-11, res_now = 1.66827e-09, res_old = 6.28967e-09
        Core 4
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 8 iterations with relres 1.71136e-09 (from 0.396932)
                Time needed 29.295 ms
                dx = 2.65786e-11, res_now = 1.71136e-09, res_old = 4.45358e-09
        Core 5
                Choosing iterative solver (local size 135424)...
                Finished with flag 1 after 3 iterations with relres 1.67653e-09 (from 0.414887)
                Time needed 16.7 ms
                dx = 3.12342e-12, res_now = 1.67653e-09, res_old = 4.26084e-09
        Core 6
                Choosing iterative solver (local size 129536)...
                Finished with flag 1 after 5 iterations with relres 1.70254e-09 (from 0.576411)
                Time needed 16.898 ms
                dx = 1.80754e-11, res_now = 1.70254e-09, res_old = 3.06685e-09
        Core 7
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.58989e-09 (from 0.795746)
                Time needed 0.643 ms
                dx = 8.85293e-13, res_now = 1.58989e-09, res_old = 2.22152e-09
Solution rank [ 1 22 24 23 23 23 23 22 1 ]
Maxres 7.86136e-09
Time 273.108 ms

Starting sweep 15 (last one)
        Core 0
                Choosing iterative solver (local size 5632)...
                Finished with flag 1 after 1 iterations with relres 1.25467e-09 (from 0.699452)
                Time needed 0.886 ms
                dx = 1.09686e-12, res_now = 1.25467e-09, res_old = 2.52736e-09
        Core 1
                Choosing iterative solver (local size 110592)...
                Finished with flag 1 after 2 iterations with relres 1.65754e-09 (from 0.374358)
                Time needed 5.984 ms
                dx = 2.46619e-12, res_now = 1.65754e-09, res_old = 4.72213e-09
        Core 2
                Choosing iterative solver (local size 117760)...
                Finished with flag 1 after 2 iterations with relres 1.7403e-09 (from 0.305758)
                Time needed 10.88 ms
                dx = 3.12012e-12, res_now = 1.7403e-09, res_old = 5.78158e-09
        Core 3
                Choosing iterative solver (local size 111872)...
                Finished with flag 1 after 3 iterations with relres 1.62722e-09 (from 0.373588)
                Time needed 12.919 ms
                dx = 4.35216e-12, res_now = 1.62722e-09, res_old = 4.73186e-09
        Core 4
                Choosing iterative solver (local size 111872)...
                Finished with flag 1 after 2 iterations with relres 1.74943e-09 (from 0.498912)
                Time needed 11.013 ms
                dx = 2.60875e-12, res_now = 1.74943e-09, res_old = 3.54324e-09
        Core 5
                Choosing iterative solver (local size 111872)...
                Finished with flag 1 after 1 iterations with relres 8.40839e-10 (from 0.843821)
                Time needed 9.149 ms
                dx = 9.07152e-13, res_now = 8.40839e-10, res_old = 2.09495e-09
        Core 6
                Choosing iterative solver (local size 107008)...
                Finished with flag 1 after 1 iterations with relres 1.44182e-09 (from 1.16718)
                Time needed 9.009 ms
                dx = 6.40839e-13, res_now = 1.44182e-09, res_old = 1.51456e-09
        Core 7
                Choosing iterative solver (local size 4608)...
                Finished with flag 1 after 1 iterations with relres 8.45485e-10 (from 2.0333)
                Time needed 0.631 ms
                dx = 5.0725e-13, res_now = 8.45485e-10, res_old = 8.69406e-10
Solution rank [ 1 18 20 19 19 19 19 18 1 ]
Maxres 5.78158e-09
Time 159.09 ms

Finished after 15 sweeps and 14.2404 seconds

Relative residual:  tensor(5.0368e-09, dtype=torch.float64)
Solver time:  0:00:14.255007