Memory Issues

[oguzhannysr]

@MuellerSeb Hello, I want to use your kriging interpolation method but I'm getting a memory error. How can I overcome this? However, although I waited for the same data in the SmartMap plugin in QGIS, I got a result. Is there anything I can change or suggest I make?

import numpy as np
import geopandas as gpd
from pykrige.ok import OrdinaryKriging
import matplotlib.pyplot as plt

gdf = gpd.read_file(r"test_EPSG4326.geojson")  

X = np.array([gdf.geometry.x, gdf.geometry.y]).T
y = gdf['m'].values

# Ordinary Kriging 
OK = OrdinaryKriging(X[:, 0], X[:, 1], y, variogram_model='linear', verbose=True, enable_plotting=False)

# İnterpolasyon
gridx = np.linspace(X[:, 0].min(), X[:, 0].max(), 100)
gridy = np.linspace(X[:, 1].min(), X[:, 1].max(), 100)

# Grid 
grid_z, ss = OK.execute('grid', gridx, gridy)


plt.figure(figsize=(10, 6))
plt.contourf(gridx, gridy, grid_z, cmap='coolwarm')
plt.scatter(X[:, 0], X[:, 1], c=y, edgecolor='k', cmap='coolwarm')
plt.colorbar(label='Maglev Değeri')
plt.title('Ordinary Kriging İnterpolasyonu')
plt.xlabel('X Koordinatı')
plt.ylabel('Y Koordinatı')
plt.show()

[mihtm13]

Hello,
in my case adding the parameter OK.execute('grid', gridx, gridy, backend='loop') or OK.execute('grid', gridx, gridy, backend='C')
helped to reduce the ammount of RAM needed. But with using the loop-backend the processing time increases.
Here is the explanation for the "backend" parameter:
backend : str, optional Specifies which approach to use in kriging. Specifying 'vectorized' will solve the entire kriging problem at once in a vectorized operation. This approach is faster but also can consume a significant amount of memory for large grids and/or large datasets. Specifying 'loop' will loop through each point at which the kriging system is to be solved. This approach is slower but also less memory-intensive. Default is 'vectorized'. Note that Cython backend is not supported for UK.

But there is another problem associated to this error, because you're not able to control which datatype, like numpy.float32, the Pykrige program is using. It seems to always default to numpy.float64, which is in most cases too memory expensive.

[oguzhannysr]

@mihtm13 , Thanks but I couldn't get to that line because I get an error on the line below.

OK = OrdinaryKriging(X[:, 0], X[:, 1], y, variogram_model='linear', verbose=True, enable_plotting=False)

[mihtm13]

After some digging there seems to be an issue with the "ok.py", where for most Numpy functions inside the "OrdinaryKriging" class no dtype is defined. Numpy will then default to numpy.float64 which increases memory consumption drastically.

If there would be a way of using numpy.float32 you'll be able to half the memory cunsumption.

I think this cloud be a new feature, to be able to define the dtype in the __init__ function of the "OrdinaryKriging" class in "ok.py".
Or in another way to add a downcast functionality for getting the lowest possible precision numerical dtype, like in pandas.