D2L - 9. Recurrent Neural Networks

https://d2l.ai/chapter_recurrent-neural-networks/index.html

9. Recurrent Neural Networks — Dive into Deep Learning 1.0.0-beta0 documentation

 

9. Recurrent Neural Networks

 

Up until now, we have focused primarily on fixed-length data. When introducing linear and logistic regression in Section 3 and Section 4 and multilayer perceptrons in Section 5, we were happy to assume that each feature vector x1 consisted of a fixed number of components x1,…,xd, where each numerical feature dj corresponded to a particular attribute. These datasets are sometimes called tabular, because they can be arranged in tables, where each example i gets its own row, and each attribute gets its own column. Crucially, with tabular data, we seldom assume any particular structure over the columns.

 

지금까지는 주로 fixed-length data에 중점을 두었습니다. 섹션 3과 섹션 4에서 선형 및 로지스틱 회귀를 소개하고 섹션 5에서 다층 퍼셉트론을 소개할 때 각 특징 벡터 x1이 고정된 수의 구성 요소 x1,....,xd로 구성되어 있다고 가정하게 되어 happy 했습니다. 여기서 각 수치 특징 xj는 특정 속성에 해당합니다. .
이러한 데이터 세트는 테이블로 정렬할 수 있기 때문에 테이블 형식이라고도 합니다. 여기서 각 example  i는 고유한 행을 갖고 각 속성은 고유한 열을 갖습니다. 결정적으로 테이블 형식 데이터의 경우 열에 대한 특정 구조를 거의 가정하지 않습니다.

 

Subsequently, in Section 7, we moved on to image data, where inputs consist of the raw pixel values at each coordinate in an image. Image data hardly fit the bill of a protypical tabular dataset. There, we needed to call upon convolutional neural networks (CNNs) to handle the hierarchical structure and invariances. However, our data were still of fixed length. Every Fashion-MNIST image is represented as a 28×28 grid of pixel values. Moreover, our goal was to develop a model that looked at just one image and then output a single prediction. But what should we do when faced with a sequence of images, as in a video, or when tasked with producing a sequentially structured prediction, as in the case of image captioning?

 

이어서 섹션 7에서 입력이 이미지의 각 좌표에서 원시 픽셀 값으로 구성되는 이미지 데이터로 이동했습니다. 이미지 데이터는 원형 테이블 형식 데이터 세트의 계산서에 거의 맞지 않습니다. 그곳에서 우리는 계층적 구조와 불변성을 처리하기 위해 CNN(컨볼루션 신경망)을 호출해야 했습니다. 그러나 데이터는 여전히 고정 길이였습니다. 모든 Fashion-MNIST 이미지는 픽셀 값의 28×28 그리드로 표현됩니다. 또한 우리의 목표는 단 하나의 이미지를 보고 단일 예측을 출력하는 모델을 개발하는 것이었습니다. 그러나 비디오에서와 같이 일련의 이미지에 직면하거나 이미지 캡션의 경우에서와 같이 순차적으로 구조화된 예측을 생성해야 하는 경우 어떻게 해야 합니까?

 

Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that models produce outputs consisting of sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that are sequences. These demands often arise simultaneously: tasks such as translating passages of text from one natural language to another, engaging in dialogue, or controlling a robot, demand that models both ingest and output sequentially-structured data.

 

수많은 학습 작업에는 sequential  데이터를 처리해야 합니다. 이미지 캡션, 음성 합성 및 음악 생성은 모두 모델이 시퀀스로 구성된 출력을 생성해야 합니다. 시계열 예측, 비디오 분석 및 음악 정보 검색과 같은 다른 영역에서 모델은 시퀀스인 입력에서 학습해야 합니다. 이러한 요구 사항은 종종 동시에 발생합니다. 하나의 자연 언어에서 다른 자연 언어로 텍스트 구절을 번역하거나, 대화에 참여하거나, 로봇을 제어하는 것과 같은 작업은 순차적으로 구조화된 데이터를 수집하고 출력하는 모델을 요구합니다.

 

Recurrent neural networks (RNNs) are deep learning models that capture the dynamics of sequences via recurrent connections, which can be thought of as cycles in the network of nodes. This might seem counterintuitive at first. After all, it is the feedforward nature of neural networks that makes the order of computation unambiguous. However, recurrent edges are defined in a precise way that ensures that no such ambiguity can arise. Recurrent neural networks are unrolled across time steps (or sequence steps), with the same underlying parameters applied at each step. While the standard connections are applied synchronously to propagate each layer’s activations to the subsequent layer at the same time step, the recurrent connections are dynamic, passing information across adjacent time steps. As the unfolded view in Fig. 9.1 reveals, RNNs can be thought of as feedforward neural networks where each layer’s parameters (both conventional and recurrent) are shared across time steps.

 

순환 신경망(RNN)은 노드 네트워크의 주기로 생각할 수 있는 recurrent  연결을 통해 시퀀스의 역학을 캡처하는 딥 러닝 모델입니다. 이것은 처음에는 직관에 반하는 것처럼 보일 수 있습니다. 결국 계산 순서를 모호하지 않게 만드는 것은 신경망의 feedforward  특성입니다. 그러나 recurrent edges는 그러한 모호성이 발생하지 않도록 하는 정확한 방식으로 정의됩니다. 순환 신경망은 시간 단계(또는 시퀀스 단계)에 걸쳐 전개되며 각 단계에서 동일한 기본 매개변수가 적용됩니다. standard connections 은 동기식(synchronously )으로 적용되어 각 계층의 활성화를 at the same time step에서 후속 계층으로 전파하는 반면, recurrent connections은 dynamic이며 인접한 시간 단계에 걸쳐 정보를 전달합니다. 그림 9.1의 펼쳐진 보기에서 알 수 있듯이 RNN은 각 계층의 매개변수(기존 및 순환 모두)가 시간 단계에 걸쳐 공유되는 feedforward   신경망으로 생각할 수 있습니다.

 

Fig. 9.1  On the left recurrent connections are depicted via cyclic edges. On the right, we unfold the RNN over time steps. Here, recurrent edges span adjacent time steps, while conventional connections are computed synchronously.

 

Like neural networks more broadly, RNNs have a long discipline-spanning history, originating as models of the brain popularized by cognitive scientists and subsequently adopted as practical modeling tools employed by the machine learning community. As with deep learning more broadly, this book adopts the machine learning perspective, focusing on RNNs as practical tools which rose to popularity in the 2010s owing to breakthrough results on such diverse tasks as handwriting recognition (Graves et al., 2008), machine translation (Sutskever et al., 2014), and recognizing medical diagnoses (Lipton et al., 2016). We point the reader interested in more background material to a publicly available comprehensive review (Lipton et al., 2015). We also note that sequentiality is not unique to RNNs. For example, the CNNs that we already introduced can be adapted to handle data of varying length, e.g., images of varying resolution. Moreover, RNNs have recently ceded considerable market share to Transformer models, which will be covered in Section 11. However, RNNs rose to prominence as the default models for handling complex sequential structure in deep learning, and remain staple models for sequential modeling to this day. The stories of RNNs and of sequence modeling are inextricably linked, and this is as much a chapter about the ABCs of sequence modeling problems as it is a chapter about RNNs.

 

보다 광범위하게 신경망과 마찬가지로 RNN은 인지 과학자들이 대중화한 후 기계 학습 커뮤니티에서 사용하는 실용적인 모델링 도구로 채택된 뇌 모델에서 비롯된 긴 학문 분야에 걸친 역사를 가지고 있습니다. 더 넓은 범위에서 딥 러닝과 마찬가지로 이 책은 기계 학습 관점을 채택하고 필기 인식(Graves et al., 2008), 기계 번역과 같은 다양한 작업에 대한 획기적인 결과로 인해 2010년대에 인기를 얻은 실용적인 도구인 RNN에 초점을 맞춥니다. (Sutskever et al., 2014), 의학적 진단 인식(Lipton et al., 2016). 우리는 더 많은 배경 자료에 관심이 있는 독자에게 공개적으로 이용 가능한 포괄적인 리뷰를 알려줍니다(Lipton et al., 2015). 또한 순차성은 RNN에 고유하지 않습니다. 예를 들어 이미 소개한 CNN은 다양한 길이의 데이터(예: 다양한 해상도의 이미지)를 처리하도록 조정할 수 있습니다. 또한 RNN은 최근 Transformer 모델에 상당한 시장 점유율을 양도했으며 이에 대해서는 섹션 11에서 다룰 것입니다. 그러나 RNN은 딥 러닝에서 복잡한 순차 구조를 처리하기 위한 기본 모델로 유명해졌으며 오늘날까지 순차 모델링의 주요 모델로 남아 있습니다. . RNN과 시퀀스 모델링의 이야기는 뗄래야 뗄 수 없이 연결되어 있으며, 이것은 RNN에 대한 챕터인 만큼 시퀀스 모델링 문제의 ABC에 대한 챕터입니다.

 

One key insight paved the way for a revolution in sequence modeling. While the inputs and targets for many fundamental tasks in machine learning cannot easily be represented as fixed length vectors, they can often nevertheless be represented as varying-length sequences of fixed length vectors. For example, documents can be represented as sequences of words. Medical records can often be represented as sequences of events (encounters, medications, procedures, lab tests, diagnoses). Videos can be represented as varying-length sequences of still images.

 

한 가지 핵심 통찰력은 시퀀스 모델링의 혁명을 위한 길을 열었습니다. 기계 학습의 많은 기본 작업에 대한 입력 및 목표는 고정 길이 벡터로 쉽게 표현할 수 없지만 그럼에도 불구하고 종종 고정 길이 벡터의 가변 길이 시퀀스로 표현할 수 있습니다. 예를 들어 문서는 일련의 단어로 나타낼 수 있습니다. 의료 기록은 종종 일련의 사건(만남, 투약, 절차, 실험실 테스트, 진단)으로 표현될 수 있습니다. 비디오는 스틸 이미지의 가변 길이 시퀀스로 나타낼 수 있습니다.

 

While sequence models have popped up in countless application areas, basic research in the area has been driven predominantly by advances on core tasks in natural language processing. Thus, throughout this chapter, we will focus our exposition and examples on text data. If you get the hang of these examples, then applying these models to other data modalities should be relatively straightforward. In the next few sections, we introduce basic notation for sequences and some evaluation measures for assessing the quality of sequentially structured model outputs. Next, we discuss basic concepts of a language model and use this discussion to motivate our first RNN models. Finally, we describe the method for calculating gradients when backpropagating through RNNs and explore some challenges that are often encountered when training such networks, motivating the modern RNN architectures that will follow in Section 10.

 

시퀀스 모델이 무수한 응용 분야에서 나타났지만 해당 분야의 기본 연구는 주로 자연어 처리의 핵심 작업에 대한 발전에 의해 주도되었습니다. 따라서 이 장 전체에서 설명과 예제는 텍스트 데이터에 중점을 둘 것입니다. 이러한 예에 익숙해지면 이러한 모델을 다른 데이터 양식에 적용하는 것이 상대적으로 간단해야 합니다. 다음 몇 섹션에서는 시퀀스에 대한 기본 표기법과 순차적으로 구조화된 모델 출력의 품질을 평가하기 위한 몇 가지 평가 방법을 소개합니다. 다음으로 언어 모델의 기본 개념에 대해 논의하고 이 논의를 사용하여 첫 번째 RNN 모델에 동기를 부여합니다. 마지막으로, 우리는 RNN을 통해 역전파할 때 그래디언트를 계산하는 방법을 설명하고 그러한 네트워크를 훈련할 때 종종 직면하는 몇 가지 문제를 탐구하여 섹션 10에서 뒤따를 최신 RNN 아키텍처에 동기를 부여합니다.

 

https://youtu.be/PahF2hZM6cs

https://youtu.be/NHb6jNH2YTg

https://youtu.be/PahF2hZM6cs