Transformer Architecture

• Enter a Query ($Q$) for search
• Extract key information $K_i$ for each search result
• Compute how similar is the key to the query: Attention Mask
• Extract required information from the search i.e. Value $V$

Goal: Identify and attend to most important features in input

1. We want to elimintate recurrence because that what gave rise to the limitations. So, we need to encode position information

   

   Figure 2: Position-Aware Encoding (@ 0:48:32)

2. Extract, query, key, value for search
   • Multiply the positional encoding with three matrices to get query, key and value encoding for each word
3. Compute attention weighting (A matix of post-softmax attention scores)

   • Compute pairwise similarity between each query and key => Dot Product (0:51:01)

     Attention Score = $\frac{Q.K^T}{scaling}$

   • Apply softmax to the attention score to get value in $[0,1]$
4. Extract features with high attention: Multiply attention weighting with Value.