Problem

Your notebook running on Databricks Runtime version 16.4 LTS fails with the following error.

RuntimeError: Failed to import <model-name> because of the following error (look up to see its traceback):

Failed to import transformers-generation.utils because of the following error (look up to see its traceback): module 'numpy._core' has no attribute 'multiarray'
File /databricks/python/lib/python3.12/site-packages/transformers/utils/import_utils.py:1976, in _LazyModule,_get module(self, module_name)
1975 try:
-> 1976
return importlib. import module("." + module_name, self._ name_)
1977 except Exception as e:

Cause

There is a mismatch between the NumPy library version on the cluster and the NumPy version required for Databricks Runtime 16.4 LTS. 

Databricks Runtime 16.4 LTS contains NumPy version 1.26.4. When you install a third party library that has a NumPy dependency on the cluster, NumPy 2.x by default is included in the install. This NumPy change occurs whether you add the library in the cluster Libraries tab, or by using the pip install command in a notebook.

You can check the driver log to see confirmation that NumPy is already present.

Requirement already satisfied: numpy in /local_disk0/.ephemeral_nfs/envs/<your-python-environment>/lib/python3.12/site-packages (from <library>) (2.x.x)

Solution

Pin the numpy==1.26.4 version to overwrite the higher NumPy versions automatically installed with the third party library. The procedure differs depending on your chosen third party library install method. 

If you use the cluster Libraries tab

Pin the numpy==1.26.4 version using the cluster settings UI. 

1. Login to your workspace.
2. Click Compute in the sidebar nav.
3. Select the cluster.
4. Navigate to the Libraries tab. 
5. Click Install New.
6. Select type PyPI and input the following command in the Package option.

      numpy==1.26.4

7. Click Install.
8. Select Restart to restart the cluster.

If you use the pip install command in a notebook

Pin the Databricks Runtime 16.4 LTS preferred NumPy version directly in pip install. 

pip install numpy==1.26.4

Restart the kernel using the "dbutils.library.restartPython()" command for the change to take effect.