Numpy Vectorization using too much memory

Post author: Yuan Liu
Post link: <a href="http://example.com/2021/02/23/2021-02-23-numpy-vectorization-using-too-much-memory/" title="Numpy Vectorization using too much memory">http://example.com/2021/02/23/2021-02-23-numpy-vectorization-using-too-much-memory/
Copyright Notice: All articles in this blog are licensed under <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/" rel="noopener" target="_blank"> BY-NC-SA unless stating additionally.

Posted on 2021-02-23 Edited on 2022-05-07

Numpy vectorization

Vectorization is a technique to accelerate computing.
numpy vectorize v1.20
However, the data type of the output of vectorized is determined by calling the function with the first element of the input.

A thread in stackoverflow descripes this problem. Briefly, each value is evaluated as an object consuming much memory.
How to avoid enormous additional memory consumption when using numpy vectorize?

I encountered this problems when I applied this techinque. It consumes all my memory (128GB).
I tried to expand my swap to 256GB. The disk is not solid state, and thus the swap is pretty slow.