Problem

The Databricks web terminal offers an interactive CLI for running shell commands. The web terminal service is proxied on port 7681 on the cluster’s Apache Spark driver. Databricks Container Services (DCS) lets you specify a Docker image when you create a cluster.

However, you notice when you use a custom Docker image, the web terminal is disabled by default.

Cause

Enabling DCS will disable the web terminal on the cluster. Access to the Workspace File System is not permitted from the web terminal.

Solution

To access the web terminal on a DCS cluster, you can use the ttyd package in your Dockerfile and launch it over port 7681. ttyd is a lightweight, web-based terminal emulator.

1. Add the ttyd package to your Dockerfile. This is a sample Dockerfile that installs the package along with its dependencies.

# Use the databricksruntime/standard:11.3-LTS image from as a sample base image
FROM databricksruntime/standard:11.3-LTS

# Install required dependencies
RUN apt-get update && apt-get install -y \
   openssh-server gnupg2 build-essential cmake wget git zlib1g-dev libuv1-dev \
   libwebsockets-dev libssl-dev libjson-c-dev acl \
   && apt-get clean \
   && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

# Download and compile ttyd from source
RUN wget https://github.com/tsl0922/ttyd/archive/refs/tags/1.6.3.tar.gz \
   && tar -xvf 1.6.3.tar.gz \
   && cd ttyd-1.6.3 \
   && mkdir build \
   && cd build \
   && cmake .. \
   && make \
   && make install \
   && cd ../.. \
   && rm -rf ttyd-1.6.3 1.6.3.tar.gz

# Set the default command to run ttyd with /bin/bash
CMD ["ttyd", "/bin/bash"]

2. Navigate to your workspace's home folder.
3. Click Create and select File.
4. Name the file web-terminal.sh and add the following shell script contents.

#!/bin/bash

# Wait for the cluster to come up
sleep 30

# Run ttyd in the background
ttyd -p 7681 bash &

5. Configure the cluster accordingly with the Docker image URL. This can be done using the UI or the API.

To launch from the UI:

1. Navigate to the Create compute page, under Access mode select Dedicated.
2. Specify a Databricks Runtime version that supports DCS.
3. Under Advanced options, select the Docker tab.
4. In the Docker image URL field, enter your custom Docker image with the appropriate authentication type.
5. Under the Init Scripts > Source tab, select Workspace and provide the file path to the init script created in the previous section.
6. Start the cluster. After the cluster spins up in the UI, navigate to the Apps tab, click the Web Terminal button. This launches the web terminal. 

To use the API, refer to the Create new cluster (AWS | Azure) API documentation. 

For more information, refer to the Customize containers with Databricks Container Service (AWS | Azure) documentation.