Note
Run this notebook in Google Colab or see the source on GitHub.
GPU acceleration
The package torchtt can use the built-in GPU acceleration from pytorch. The tensor can be moved to the desired device using the .to() method.
Imports and check if any CUDA device is available.
[1]:
import time
import torch as tn
try:
import torchtt as tntt
except:
print('Installing torchTT...')
%pip install git+https://github.com/ion-g-ion/torchTT
import torchtt as tntt
print('CUDA available:',tn.cuda.is_available())
print('Device name: ' + tn.cuda.get_device_name())
CUDA available: True
Device name: NVIDIA GeForce RTX 4090 Laptop GPU
Define a function to test. It performs 2 matrix vector products in the tensor-train (TT) format and a rank rounding. The return result is a scalar.
[2]:
def f(x,A,y):
"""
fonction that performs operations of tensors in TT.
Args:
x (tnt.TT): input TT tensor
A (tnt.TT): input TT matrix
y (tnt.TT): input TT tensor
Returns:
torch.tensor: result
"""
z = A @ y + A @ y # operatio that grows the rank
z = z.round(1e-12) # rank rounding (contains QR and SVD decomposition)
z += z+x # some other operation
return tntt.dot(x,z) # contract the tensor
Generate random tensors in the TT-format (on the CPU).
[3]:
x = tntt.random([200,300,400,500],[1,8,8,8,1])
y = tntt.random([200,300,400,500],[1,8,8,8,1])
A = tntt.random([(200,200),(300,300),(400,400),(500,500)],[1,8,8,8,1])
Run the function f() once to warm up, then report the average time over five runs using time.perf_counter().
[4]:
n_runs = 5
f(x,A,y) # warm up
tme_cpu = time.perf_counter()
for _ in range(n_runs):
f(x,A,y)
tme_cpu = (time.perf_counter() - tme_cpu) / n_runs
print('Average time on CPU: ',tme_cpu,' seconds')
Average time on CPU: 0.828291226598958 seconds
Move the defined tensors on GPU. Similarily to pytorch tensors one can use the function cuda() to return a copy of a TT instance on the GPU. All the cores of the returned TT object are on the GPU.
[5]:
cuda_name = 'cuda:0'
x = x.to(cuda_name)
y = y.to(cuda_name)
A = A.to(cuda_name)
The function is executed once without timing to warm up CUDA. The synchronization before the timer below waits for the warm-up computation to finish.
[6]:
result_gpu = f(x,A,y)
Run the function again and report the average time over the same number of runs. Synchronize CUDA once before starting the timer and at the end of each loop iteration so each GPU computation finishes before the next run starts. Divide the elapsed time by the number of runs. Input transfers, warm-up, and copying the final result to CPU are outside the timer.
[7]:
tn.cuda.synchronize(cuda_name)
tme_gpu = time.perf_counter()
for _ in range(n_runs):
result_gpu = f(x,A,y)
tn.cuda.synchronize(cuda_name)
tme_gpu = (time.perf_counter() - tme_gpu) / n_runs
result_cpu = result_gpu.cpu()
print('Average time with CUDA: ',tme_gpu,' seconds')
Average time with CUDA: 0.16208719659916823 seconds
The speedup is reported:
[8]:
print('Speedup: ',tme_cpu/tme_gpu,' times.')
Speedup: 5.1101582603545905 times.
This time we perform the same test without using the rank rounding. The expected result is better since the rank rounding contains QR decompositions and singular value decompositions (SVD), which are not that parallelizable. Both devices are warmed up before timing, and the average times are measured as above. CUDA is synchronized once before the loop and at the end of each iteration, and the final result is copied to CPU after timing.
[9]:
def g(x,A,y):
"""
fonction that performs operations of tensors in TT.
Args:
x (tnt.TT): input TT tensor
A (tnt.TT): input TT matrix
y (tnt.TT): input TT tensor
Returns:
torch.tensor: result
"""
z = A @ y + A @ y # operatio that grows the rank
z += z+x # some other operation
return tntt.dot(x,z) # contract the tensor
[10]:
# put tensors on CPU
x, y, A = x.cpu(), y.cpu(), A.cpu()
# perform the test
g(x,A,y) # warm up
tme_cpu = time.perf_counter()
for _ in range(n_runs):
g(x,A,y)
tme_cpu = (time.perf_counter() - tme_cpu) / n_runs
print('Average time on CPU: ',tme_cpu,' seconds')
# move the tensors back to GPU
x, y, A = x.to(cuda_name), y.cuda(cuda_name), A.cuda(cuda_name)
# execute the function
result_gpu = g(x,A,y) # warm up
tn.cuda.synchronize(cuda_name)
tme_gpu = time.perf_counter()
for _ in range(n_runs):
result_gpu = g(x,A,y)
tn.cuda.synchronize(cuda_name)
tme_gpu = (time.perf_counter() - tme_gpu) / n_runs
result_cpu = result_gpu.cpu()
print('Average time with CUDA: ',tme_gpu,' seconds')
print('Speedup: ',tme_cpu/tme_gpu,' times.')
Average time on CPU: 0.5256847355994978 seconds
Average time with CUDA: 0.021357184399676044 seconds
Speedup: 24.613953120499893 times.
A tensor can be copied to a differenct device using the torchtt.TT.to() method. Usage is similar to torch.tensor.to().
[11]:
dev = tn.cuda.current_device()
x_cuda = x.to(dev)
x_cpu = x_cuda.to(None)