Quiz: Attention Mechanisms: Teaching Neural Networks Where to Look

Question 1 of 7

What is the fundamental problem with RNNs that attention mechanisms solve?

They process sequentially and struggle to remember information from many steps back

They require too much memory to store embeddings

They cannot process individual words accurately

They only work with English language inputs

Question 2 of 7

In the translation example, how many numbers does the RNN encoder compress 12 words into?

512 numbers

768 numbers

1024 numbers

256 numbers

Question 3 of 7

What is the key difference between embeddings and hidden state vectors?

Embeddings are static while hidden states update dynamically as the RNN reads each word

Embeddings use 512 dimensions while hidden states use 768 dimensions

Embeddings work for translation while hidden states work for search

Embeddings are computed in parallel while hidden states are sequential

Question 4 of 7

What mathematical operation is used to measure similarity between Query and Keys in the attention mechanism?

Dot product

Cross product

Matrix multiplication

Cosine similarity

Question 5 of 7

What function converts attention scores into weights that sum to 1.0?

Softmax

ReLU

Sigmoid

Tanh

Question 6 of 7

How much faster is attention compared to RNNs when training on 1000 words?

100x faster

50x faster

1000x faster

10x faster

Question 7 of 7

How many attention heads do transformers typically use per layer?

8-12 heads

4-6 heads

16-20 heads

2-4 heads