Problem

You are trying to access the metadata server for your EC2 instance but cannot connect.

Cause

This is the intended, default behavior. It is functioning as designed.

Solution

Use an init script to apply a custom iptables configuration to your Databricks cluster which enables access to the metadata server.

1. Define a location to store the init script. If you do not already have a folder for your init script, you must create one. For example, using dbfs:/databricks/<init-script-folder>:

   %scala
   
   dbutils.fs.mkdirs("dbfs:/databricks/<init-script-folder>/")

2. Create the init script:

   %scala
   
   dbutils.fs.put("dbfs:/databricks/<init-script-folder>/iptables.sh","""
   #!/bin/bash 
   sudo iptables -A INPUT -s 169.254.169.254 -j ACCEPT 
   sudo iptables -A OUTPUT -d 169.254.169.254 -j ACCEPT
   """,True)

3. Verify that the init script was created on your cluster:

   %scala
   
   display(dbutils.fs.ls("dbfs:/databricks/<init-script-folder>/iptables.sh"))

4. Configure the init script as a cluster-scoped init script on your cluster.
5. Restart your cluster. 

After the cluster restarts, the init script takes effect. You now have access to the metadata server for your EC2 instance.