path: root/Enhancing_Data_Science_Outcomes_With_Efficient_Workflow/2_04_embedding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0bf7f930-76a1-4c16-84e4-cf1e73b54c55",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/dli\"> <img src=\"images/DLI_Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "400a41da-bc38-4e9a-9ece-d2744ffb16b0",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Enhancing Data Science Outcomes With Efficient Workflow #"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8897c66c-4f9d-48b4-a60b-ddae16f2f61b",
   "metadata": {},
   "source": [
    "## 04 - Embeddings ##\n",
    "In this lab, you will use high-performance computing to create machine learning solutions. This lab covers the model development portion of the data science workflow. A good machine learning solution excels that both accuracy and inference performance. \n",
    "\n",
    "<p><img src='images/pipeline_overview_2.png' width=1080></p>\n",
    "\n",
    "**Table of Contents**\n",
    "<br>\n",
    "This notebook covers the below sections: \n",
    "1. [Entity Embedding](#s4-1)\n",
    "2. [Training the Embeddings](#s4-2)\n",
    "    * [Preparing the Data - Normalization](#s4-2.1)\n",
    "    * [Model Building](#s4-2.2)\n",
    "    * [Being Training](#s4-2.3)\n",
    "3. [Visualizing the Embeddings](#s4-3)\n",
    "4. [Conclusion](#s4-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28538773-6b95-4840-aca2-73a6f7d98b07",
   "metadata": {},
   "source": [
    "<a name='s4-1'></a>\n",
    "## Entity Embeddings ##\n",
    "[Entity Embeddings](https://arxiv.org/pdf/1604.06737.pdf) are very similar to word embeddings used in NLP. They are a way to represent categorical features in a defined latent space. In the latent space, categories that are semantically similar have similar vectors. Embeddings can be trained to assign a learnable feature vector to each category. Using embeddings, each categorical value is mapped to its own associated vector representation that is more informative than a single point value. Even though embeddings require a large amount of data and computational resources to train, they have proven to be a great alternative encoding method to consider. Once trained, embeddings can boost the performance of downstream machine learning tasks when used as the input features. Users can combine the power of deep learning with traditional machine learning on tabular data. \n",
    "\n",
    "<p><img src='images/embedding.png' width=720></p>\n",
    "\n",
    "Reasons for using embeddings include: \n",
    "* It is much more efficient than the one-hot approach for encoding when cardinality if high\n",
    "* Allows rich relationships and complexities between categories to be captured\n",
    "* Reduce memory usage and speed up downstream machine learning model training\n",
    "* Once trained, the same embedding can be used for various use cases\n",
    "* Can be used to visualize categorical data and for data clustering, since the embedding space quantifies semantic similarity as distance between the categories in the latent space\n",
    "* Mitigates the need to perform cumbersome manual feature engineering, which requires extensive domain knowledge\n",
    "\n",
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "Below are some tips about embeddings: \n",
    "* Requires training with large amounts of data, making it inappropriate for unseen data such as when new categories are added\n",
    "* Can overfit\n",
    "* Difficult to interpret"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ba4160d-4b41-40d3-93bc-f1fae0b9dddc",
   "metadata": {},
   "source": [
    "<a name='s4-2'></a>\n",
    "## Training the Embeddings ##\n",
    "Embeddings aim to represent each entity as a numeric vector such that products in similar context have similar vectors. Mathematically, similar entities will have a large dot product whereas every entity when one-hot encoded has a zero dot product with every other entity. This is because all one-hot vectors are orthogonal. \n",
    "\n",
    "We will use [PyTorch](https://pytorch.org/) to train a simple fully-connected neural network. A surrogate problem is setup for the purpose of finding the embedding vectors. Neural networks have difficultly with sparse categorical features. Traditionally, embeddings are a way to reduce those features to increase model performance. \n",
    "\n",
    "Technically, the idea of an embedding layer is very similar to a dense or linear layer (without bias) in the neural network. When training an embedding this way, users will one-hot encode the categorical data so each record becomes a vector with C features, where C is the cardinality. We then perform matrix vector multiplication on the input vector and the weights before feeding the next layer. This is inefficient when the number of input features is large and sparse, as is the case for categorical features from a tabular dataset. \n",
    "\n",
    "A better and more efficient approach would be to train a `torch.nn.Embedding` layer, which can be treated as a \"lookup\" table with the label-encoded category id as the index. By using choosing this, we avoid one-hot encoding and the matrix vector multiplication. \n",
    "\n",
    "<p><img src='images/surrogate_problem.png' width=720></p>\n",
    "\n",
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "Embeddings will naturally be affected by how the surrogate problem is defined. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ec50a570-247f-4cfc-8dc5-2c2b501de703",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# import dependencies\n",
    "from tqdm import tqdm\n",
    "import cudf\n",
    "import cuml\n",
    "import dask_cudf\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as torch_optim\n",
    "from torch.utils.data import Dataset, DataLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "036bf6ee-d5cb-4f20-a591-681706a098ac",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# set device cuda to use GPU\n",
    "device=torch.device('cuda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3726fc69-2a2b-42e2-be12-d235ce2322c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# define features and label\n",
    "cols=['brand', 'cat_0', 'cat_1', 'cat_2', 'price', 'target']\n",
    "cat_cols=['brand', 'cat_0', 'cat_1', 'cat_2']\n",
    "label='target'\n",
    "\n",
    "feature_cols=[col for col in cols if col != label]\n",
    "cont_cols=[col for col in feature_cols if col not in cat_cols] # ['price']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ae87d23f-0c67-4758-8842-ca5770e740f9",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total of 2461697 records.\n"
     ]
    }
   ],
   "source": [
    "# read data\n",
    "parquet_dir='processed_parquet'\n",
    "\n",
    "ddf=dask_cudf.read_parquet(parquet_dir, columns=cols)\n",
    "gdf=ddf.compute()\n",
    "\n",
    "print(f'Total of {len(gdf)} records.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9110c9d-5924-4cb2-8bf3-cabd398aad0e",
   "metadata": {},
   "source": [
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "Even though we intend to keep all the data in one GPU, we still recommend loading data with `Dask-cuDF`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f782bc7e-e6c4-4d87-a839-5a99227dca7c",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>brand</th>\n",
       "      <th>cat_0</th>\n",
       "      <th>cat_1</th>\n",
       "      <th>cat_2</th>\n",
       "      <th>price</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>100.229996</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>871.839966</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>872.090027</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>306.690002</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>13</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>24</td>\n",
       "      <td>334.349976</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   brand  cat_0  cat_1  cat_2       price  target\n",
       "0      1      6      5      2  100.229996       1\n",
       "1      2      1      1      1  871.839966       1\n",
       "2      2      1      1      1  872.090027       1\n",
       "3      2      6      5      2  306.690002       1\n",
       "4     13      2      3     24  334.349976       1"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gdf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3673f202-7aea-43a7-a569-4c210a614529",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'brand': (3303, 7), 'cat_0': (14, 3), 'cat_1': (61, 3), 'cat_2': (90, 3)}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the embedding vectors will start with 0 so we decrease the categorical values by 1 to match\n",
    "gdf[cat_cols]=gdf[cat_cols]-1\n",
    "\n",
    "n_uniques=gdf.nunique()\n",
    "\n",
    "# use higher of 4th root of nunique and 3 for vector dimension\n",
    "embedding_sizes={col: (n_uniques[col], max(3, int(n_uniques[col]**0.25))) for col in cat_cols}\n",
    "embedding_sizes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a327c1f9-0683-45f1-90a6-6d4d4daa093c",
   "metadata": {
    "tags": []
   },
   "source": [
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "The size of embeddings can become very large. For example, large embeddings are usually needed for users and items for large platforms. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c1c7fee-dad0-4009-a55c-513465db8a7c",
   "metadata": {},
   "source": [
    "<a name='s4-2.1'></a>\n",
    "### Preparing the Data - Normalization ###\n",
    "**Normalization** is required to enable neural networks to leverage numerical features. Tree-based models do not require normalization as they define the split independent of the scale of a feature. Without normalization, neural networks are difficult to train. The reason is that different numerical features have different scales. When we combine the features in a hidden layer, the different scales make it more difficult to extract patterns from it. \n",
    "\n",
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "We will also implement a `torch.nn.BatchNorm1d`[[doc]](https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm1d.html) layer to mitigate the exploding gradient problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fb1840b3-a7d8-4b91-98ef-bddf59afd5e6",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# normalize data\n",
    "gdf['price']=cuml.preprocessing.StandardScaler().fit_transform(gdf[['price']])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6991948-f79a-4b51-b3a9-2571b2be5262",
   "metadata": {
    "tags": []
   },
   "source": [
    "<a name='s4-2.2'></a>\n",
    "### Model Building ###\n",
    "We construct a model with several layers. The embeddings will be the same dimension as num_unique x vector_size. The embeddings will be concatenated, along with the continous variable(s), before they are fed into the next layer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "35a8055b-8b7b-4fb8-8d3a-9f36fc03b171",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# define neural network with embedding layers\n",
    "class ProductPurchaseModel(nn.Module):\n",
    "    def __init__(self, embedding_sizes, n_cont):\n",
    "        super().__init__()\n",
    "        # make an embedding for each categorical feature\n",
    "        # The `nn.Embedding` layer can be thought of as a lookup table where the key is \n",
    "        # the category index and the value is the corresponding embedding vector\n",
    "        self.embeddings=nn.ModuleList([nn.Embedding(n_categories, size) for n_categories, size in embedding_sizes.values()])\n",
    "        \n",
    "        # n_emb is the length of all embeddings combined\n",
    "        n_emb=sum(e.embedding_dim for e in self.embeddings)\n",
    "        \n",
    "        self.n_emb=n_emb\n",
    "        self.n_cont=n_cont\n",
    "        self.emb_drop = nn.Dropout(0.6)\n",
    "        \n",
    "        # apply dropout, batch norm and linear layers\n",
    "        self.bn1=nn.BatchNorm1d(self.n_cont)\n",
    "        self.lin1=nn.Linear(self.n_emb + self.n_cont, 200)\n",
    "        self.drop1=nn.Dropout(0.3)\n",
    "        self.bn2=nn.BatchNorm1d(200)\n",
    "        self.drop2=nn.Dropout(0.3)\n",
    "        self.lin2=nn.Linear(200, 70)\n",
    "        self.bn3=nn.BatchNorm1d(70)\n",
    "        self.lin3=nn.Linear(70, 2)\n",
    "\n",
    "    def forward(self, X_cat, X_cont):\n",
    "        # map each categorical feature to the embedding vector on its corresponding embedding layer\n",
    "        x_1=[embedding(X_cat[:, idx]) for idx, embedding in enumerate(self.embeddings)]\n",
    "        \n",
    "        # concatenate all categorical embedding vectors together\n",
    "        x_1=torch.cat(x_1, 1)\n",
    "        \n",
    "        # apply random drop out, normalization, and activation\n",
    "        x_1=self.emb_drop(x_1)\n",
    "        x_2=self.bn1(X_cont)\n",
    "        \n",
    "        # concatenate categorical embeddings to input layer from continuous variable(s)\n",
    "        x_1=torch.cat([x_1, x_2], 1)\n",
    "        \n",
    "        # apply random drop out, normalization, and activation\n",
    "        x_1=F.relu(self.lin1(x_1))\n",
    "        x_1=self.drop1(x_1)\n",
    "        x_1=self.bn2(x_1)\n",
    "        x_1=F.relu(self.lin2(x_1))\n",
    "        x_1=self.drop2(x_1)\n",
    "        x_1=self.bn3(x_1)\n",
    "        x_1=self.lin3(x_1)\n",
    "        return x_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c52e50a2-99b6-4a8c-aa65-5f11a7806c6e",
   "metadata": {},
   "source": [
    "<p><img src='images/tip.png' width=720></p>\n",
    "\n",
    "Tabular data uses shallow models with huge embedding tables and few feed-forward layers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "5b7d18b1-d29e-43d4-8091-3aba41968ebf",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ProductPurchaseModel(\n",
       "  (embeddings): ModuleList(\n",
       "    (0): Embedding(3303, 7)\n",
       "    (1): Embedding(14, 3)\n",
       "    (2): Embedding(61, 3)\n",
       "    (3): Embedding(90, 3)\n",
       "  )\n",
       "  (emb_drop): Dropout(p=0.6, inplace=False)\n",
       "  (bn1): BatchNorm1d(1, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (lin1): Linear(in_features=17, out_features=200, bias=True)\n",
       "  (drop1): Dropout(p=0.3, inplace=False)\n",
       "  (bn2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (drop2): Dropout(p=0.3, inplace=False)\n",
       "  (lin2): Linear(in_features=200, out_features=70, bias=True)\n",
       "  (bn3): BatchNorm1d(70, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (lin3): Linear(in_features=70, out_features=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# instantiate model\n",
    "model=ProductPurchaseModel(embedding_sizes, len(cont_cols))\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35dab8e-f1cd-484b-999e-b9e0f7e79edd",
   "metadata": {},
   "source": [
    "Next, we define a `torch.utils.data.Dataset` class to be use by `torch.utils.data.DataLoader`. The Dataset is makes it easier to track separate categorical and continuous variables. The DatalLoader wraps an iterable around the Dataset to enable easy access to the samples. More information about Dataset and DataLoader can be found in quick PyTorch [guide](https://pytorch.org/tutorials/beginner/basics/data_tutorial.html). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "98f74906-7b79-4fda-8626-df17023ee512",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# define dataset\n",
    "class myDataset(Dataset):\n",
    "    def __init__(self, X, y, cat_cols, cont_cols):\n",
    "        self.X_cat=torch.as_tensor(X.loc[:, cat_cols].copy().values.astype('int32'), device=device)\n",
    "        self.X_cont=torch.as_tensor(X.loc[:, cont_cols].copy().values.astype('float32'), device=device)\n",
    "        self.y=torch.as_tensor(y.astype('int64'), device=device)\n",
    "        \n",
    "    def __len__(self):\n",
    "        return len(self.y)\n",
    "    \n",
    "    def __getitem__(self, idx): \n",
    "        return self.X_cat[idx], self.X_cont[idx], self.y[idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a0973509-6a11-49d8-b346-ab9ec8cfaef5",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# instantiate dataset\n",
    "X_train=gdf[feature_cols]\n",
    "y_train=gdf['target'].values\n",
    "\n",
    "train_ds=myDataset(X_train, y_train, cat_cols, cont_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5336cfd0-39ed-4285-9b66-e4f5d1b7d75e",
   "metadata": {},
   "source": [
    "<a name='s4-2.3'></a>\n",
    "### Begin Training ###\n",
    "We will set some parameters for training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0604708e-1c2c-485b-a029-eadd17356a03",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# set optimizer\n",
    "def get_optimizer(model, lr = 0.001, wd = 0.0):\n",
    "    parameters=filter(lambda p: p.requires_grad, model.parameters())\n",
    "    optim=torch_optim.Adam(parameters, lr=lr, weight_decay=wd)\n",
    "    return optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39e0ce25-f65c-4330-98cc-34ee4b30bae4",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# define training function\n",
    "def train_model(model, optim, train_dl):\n",
    "    # set the model to training, which is useful for BatchNorm and Dropout layers that behave differently during training and evaluation\n",
    "    model.train()\n",
    "    total=0\n",
    "    sum_loss=0\n",
    "    \n",
    "    # iterate through batches\n",
    "    for b, (X_cat, X_cont, y) in enumerate(train_dl):\n",
    "        batch=y.shape[0]\n",
    "        \n",
    "        # forward pass\n",
    "        output=model(X_cat, X_cont)\n",
    "        \n",
    "        # calculate loss\n",
    "        loss=F.cross_entropy(output, y)\n",
    "        \n",
    "        # zero out the gradients so the parameters update correctly, otherwise gradients would be combined with old\n",
    "        optim.zero_grad()\n",
    "        loss.backward()\n",
    "        optim.step()\n",
    "        \n",
    "        # calculate total loss per batch\n",
    "        total+=batch\n",
    "        sum_loss+=batch*(loss.item())\n",
    "    return sum_loss/total"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a60dd511-3121-4eb0-beb7-3a03d56de202",
   "metadata": {},
   "source": [
    "Instantiate a `torch.utils.data.DataLoader` and begin training. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a25e4e6-f0b5-4bbc-8a1d-0eee74c7faaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define training loop\n",
    "def train_loop(model, epochs, lr=0.01, wd=0.0):\n",
    "    # instantiate optimizer\n",
    "    optim=get_optimizer(model, lr = lr, wd = wd)\n",
    "    \n",
    "    # iterate through number of epochs\n",
    "    for i in tqdm(range(epochs)): \n",
    "        loss=train_model(model, optim, train_dl)\n",
    "        print(\"training loss: \", round(loss, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68b43459-fb0a-4c13-9371-7c15327ff624",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "# define batch size and begin training\n",
    "batch_size=1000\n",
    "train_dl=DataLoader(train_ds, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "train_loop(model, epochs=3, lr=0.05, wd=0.00001)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d6656b3-3642-4279-b787-0c034c45b739",
   "metadata": {},
   "source": [
    "<a name='s4-3'></a>\n",
    "## Visualizing the Embeddings ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20973ee4-a723-4931-bf50-8efffe275026",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# visualize embeddings\n",
    "\n",
    "# import dependencies\n",
    "import plotly.express as px\n",
    "import pandas as pd\n",
    "\n",
    "# pick category to visualize\n",
    "category='brand'\n",
    "\n",
    "category_label=pd.read_parquet(f'categories/unique.{category}.parquet')[category]\n",
    "category_label=category_label[1:]\n",
    "\n",
    "embeddings_idx=list(embedding_sizes.keys()).index(category)\n",
    "embeddings=model.embeddings[embeddings_idx].weight.detach().cpu().numpy()\n",
    "\n",
    "fig=px.scatter_3d(\n",
    "    x=embeddings[:, 0], \n",
    "    y=embeddings[:, 1], \n",
    "    z=embeddings[:, 2], \n",
    "    text=category_label, \n",
    "    height=720\n",
    ")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "130a2b16-89e5-4eda-8155-014a75a3638e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# persist embeddings\n",
    "!mkdir trained_embedding_weights\n",
    "\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "for idx, each_col in enumerate(cat_cols): \n",
    "    weights=model.embeddings[idx].weight.detach().cpu().numpy()\n",
    "    pd.DataFrame(weights).to_csv(f'trained_embedding_weights/{each_col}.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc7cce0e-6dcb-4d5a-82dd-e8074abaaaec",
   "metadata": {},
   "source": [
    "<a name='s4-4'></a>\n",
    "## Conclusion ##\n",
    "Deep Learning is very good at feature extraction, which can be used for finding categorical embeddings. This is the advantage of using a Deep Learning approach, as it requires way less feature engineering and less dependent on domain knowledge. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997bd6f7-9efb-4fee-b3d4-9d4454694c7b",
   "metadata": {},
   "source": [
    "<a href=\"https://www.nvidia.com/dli\"> <img src=\"images/DLI_Header.png\" alt=\"Header\" style=\"width: 400px;\"/> </a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}