Note

Run this notebook in Google Colab or see the source on GitHub.

GPU acceleration

The package torchtt can use the built-in GPU acceleration from pytorch. The tensor can be moved to the desired device using the .to() method.

Imports and check if any CUDA device is available.

[1]:
import time
import torch as tn
try:
    import torchtt as tntt
except:
    print('Installing torchTT...')
    %pip install git+https://github.com/ion-g-ion/torchTT
    import torchtt as tntt

print('CUDA available:',tn.cuda.is_available())
print('Device name: ' + tn.cuda.get_device_name())
CUDA available: True
Device name: NVIDIA GeForce RTX 4090 Laptop GPU

Define a function to test. It performs 2 matrix vector products in the tensor-train (TT) format and a rank rounding. The return result is a scalar.

[2]:
def f(x,A,y):
    """
    fonction that performs operations of tensors in TT.

    Args:
        x (tnt.TT): input TT tensor
        A (tnt.TT): input TT matrix
        y (tnt.TT): input TT tensor

    Returns:
        torch.tensor: result
    """
    z = A @ y + A @ y # operatio that grows the rank
    z = z.round(1e-12) # rank rounding (contains QR and SVD decomposition)
    z += z+x # some other operation
    return tntt.dot(x,z) # contract the tensor

Generate random tensors in the TT-format (on the CPU).

[3]:
x = tntt.random([200,300,400,500],[1,8,8,8,1])
y = tntt.random([200,300,400,500],[1,8,8,8,1])
A = tntt.random([(200,200),(300,300),(400,400),(500,500)],[1,8,8,8,1])

Run the function f() once to warm up, then report the average time over five runs using time.perf_counter().

[4]:
n_runs = 5
f(x,A,y) # warm up
tme_cpu = time.perf_counter()
for _ in range(n_runs):
    f(x,A,y)
tme_cpu = (time.perf_counter() - tme_cpu) / n_runs
print('Average time on CPU: ',tme_cpu,' seconds')
Average time on CPU:  0.828291226598958  seconds

Move the defined tensors on GPU. Similarily to pytorch tensors one can use the function cuda() to return a copy of a TT instance on the GPU. All the cores of the returned TT object are on the GPU.

[5]:
cuda_name = 'cuda:0'
x = x.to(cuda_name)
y = y.to(cuda_name)
A = A.to(cuda_name)

The function is executed once without timing to warm up CUDA. The synchronization before the timer below waits for the warm-up computation to finish.

[6]:
result_gpu = f(x,A,y)

Run the function again and report the average time over the same number of runs. Synchronize CUDA once before starting the timer and at the end of each loop iteration so each GPU computation finishes before the next run starts. Divide the elapsed time by the number of runs. Input transfers, warm-up, and copying the final result to CPU are outside the timer.

[7]:
tn.cuda.synchronize(cuda_name)
tme_gpu = time.perf_counter()
for _ in range(n_runs):
    result_gpu = f(x,A,y)
    tn.cuda.synchronize(cuda_name)
tme_gpu = (time.perf_counter() - tme_gpu) / n_runs
result_cpu = result_gpu.cpu()
print('Average time with CUDA: ',tme_gpu,' seconds')
Average time with CUDA:  0.16208719659916823  seconds

The speedup is reported:

[8]:
print('Speedup: ',tme_cpu/tme_gpu,' times.')
Speedup:  5.1101582603545905  times.

This time we perform the same test without using the rank rounding. The expected result is better since the rank rounding contains QR decompositions and singular value decompositions (SVD), which are not that parallelizable. Both devices are warmed up before timing, and the average times are measured as above. CUDA is synchronized once before the loop and at the end of each iteration, and the final result is copied to CPU after timing.

[9]:
def g(x,A,y):
    """
    fonction that performs operations of tensors in TT.

    Args:
        x (tnt.TT): input TT tensor
        A (tnt.TT): input TT matrix
        y (tnt.TT): input TT tensor

    Returns:
        torch.tensor: result
    """
    z = A @ y + A @ y # operatio that grows the rank
    z += z+x # some other operation
    return tntt.dot(x,z) # contract the tensor
[10]:
# put tensors on CPU
x, y, A = x.cpu(), y.cpu(), A.cpu()
# perform the test
g(x,A,y) # warm up
tme_cpu = time.perf_counter()
for _ in range(n_runs):
    g(x,A,y)
tme_cpu = (time.perf_counter() - tme_cpu) / n_runs
print('Average time on CPU: ',tme_cpu,' seconds')
# move the tensors back to GPU
x, y, A = x.to(cuda_name), y.cuda(cuda_name), A.cuda(cuda_name)
# execute the function
result_gpu = g(x,A,y) # warm up
tn.cuda.synchronize(cuda_name)
tme_gpu = time.perf_counter()
for _ in range(n_runs):
    result_gpu = g(x,A,y)
    tn.cuda.synchronize(cuda_name)
tme_gpu = (time.perf_counter() - tme_gpu) / n_runs
result_cpu = result_gpu.cpu()
print('Average time with CUDA: ',tme_gpu,' seconds')

print('Speedup: ',tme_cpu/tme_gpu,' times.')
Average time on CPU:  0.5256847355994978  seconds
Average time with CUDA:  0.021357184399676044  seconds
Speedup:  24.613953120499893  times.

A tensor can be copied to a differenct device using the torchtt.TT.to() method. Usage is similar to torch.tensor.to().

[11]:
dev = tn.cuda.current_device()
x_cuda = x.to(dev)
x_cpu = x_cuda.to(None)