HF-NLP-USING 🤗 TRANSFORMERS : Introduction

https://huggingface.co/learn/nlp-course/chapter2/1?fw=pt

Introduction - Hugging Face NLP Course

 

 

As you saw in Chapter 1, Transformer models are usually very large. With millions to tens of billions of parameters, training and deploying these models is a complicated undertaking. Furthermore, with new models being released on a near-daily basis and each having its own implementation, trying them all out is no easy task.

 

1장에서 본 것처럼 Transformer 모델은 일반적으로 매우 큽니다. 수백만에서 수백억 개의 매개변수를 사용하여 이러한 모델을 교육하고 배포하는 것은 복잡한 작업입니다. 더욱이 새로운 모델이 거의 매일 출시되고 각각 자체 구현이 있기 때문에 모든 모델을 시험해 보는 것은 쉬운 일이 아닙니다.

 

The 🤗 Transformers library was created to solve this problem. Its goal is to provide a single API through which any Transformer model can be loaded, trained, and saved. The library’s main features are:

 

🤗 Transformers 라이브러리는 이 문제를 해결하기 위해 만들어졌습니다. 목표는 모든 Transformer 모델을 로드하고, 훈련하고, 저장할 수 있는 단일 API를 제공하는 것입니다. library 의 주요 기능은 다음과 같습니다.

 

• Ease of use: Downloading, loading, and using a state-of-the-art NLP model for inference can be done in just two lines of code.
  
  
• 사용 용이성: 단 두 줄의 코드로 추론을 위한 최첨단 NLP 모델을 다운로드, 로드 및 사용할 수 있습니다.
  
  
• Flexibility: At their core, all models are simple PyTorch nn.Module or TensorFlow tf.keras.Model classes and can be handled like any other models in their respective machine learning (ML) frameworks.
  
  
• 유연성: 기본적으로 모든 모델은 간단한 PyTorch nn.Module 또는 TensorFlow tf.keras.Model 클래스이며 해당 기계 학습(ML) 프레임워크에서 다른 모델처럼 처리될 수 있습니다.
• Simplicity: Hardly any abstractions are made across the library. The “All in one file” is a core concept: a model’s forward pass is entirely defined in a single file, so that the code itself is understandable and hackable.
  
  
• 단순성: 라이브러리 전반에 걸쳐 추상화가 거의 이루어지지 않습니다. "All in one file"이 핵심 개념입니다. 모델의 정방향 전달이 단일 파일에 완전히 정의되므로 코드 자체를 이해하고 해킹할 수 있습니다.
  
  

This last feature makes 🤗 Transformers quite different from other ML libraries. The models are not built on modules that are shared across files; instead, each model has its own layers. In addition to making the models more approachable and understandable, this allows you to easily experiment on one model without affecting others.

 

이 마지막 기능은 🤗 Transformer를 다른 ML 라이브러리와 상당히 다르게 만듭니다. 모델은 파일 간에 공유되는 모듈을 기반으로 구축되지 않습니다. 대신 각 모델에는 자체 레이어가 있습니다. 모델을 더욱 접근하기 쉽고 이해하기 쉽게 만드는 것 외에도 다른 모델에 영향을 주지 않고 한 모델을 쉽게 실험할 수 있습니다.

 

This chapter will begin with an end-to-end example where we use a model and a tokenizer together to replicate the pipeline() function introduced in Chapter 1. Next, we’ll discuss the model API: we’ll dive into the model and configuration classes, and show you how to load a model and how it processes numerical inputs to output predictions.

 

이 장은 모델과 토크나이저를 함께 사용하여 1장에서 소개한 파이프라인() 함수를 복제하는 엔드투엔드 예제로 시작합니다. 다음으로 모델 API에 대해 논의하겠습니다. 모델 및 구성 클래스를 자세히 살펴보고 모델을 로드하는 방법과 숫자 입력을 처리하여 예측을 출력하는 방법을 보여 드리겠습니다.

 

Then we’ll look at the tokenizer API, which is the other main component of the pipeline() function. Tokenizers take care of the first and last processing steps, handling the conversion from text to numerical inputs for the neural network, and the conversion back to text when it is needed. Finally, we’ll show you how to handle sending multiple sentences through a model in a prepared batch, then wrap it all up with a closer look at the high-level tokenizer() function.

 

그런 다음 파이프라인() 함수의 또 다른 주요 구성 요소인 토크나이저 API를 살펴보겠습니다. 토크나이저는 첫 번째와 마지막 처리 단계를 처리하여 텍스트를 신경망의 숫자 입력으로 변환하고 필요할 때 다시 텍스트로 변환하는 작업을 처리합니다. 마지막으로, 준비된 배치에서 모델을 통해 여러 문장 전송을 처리하는 방법을 보여주고, 상위 수준 tokenizer() 함수를 자세히 살펴보며 마무리하겠습니다.

 

⚠️ In order to benefit from all features available with the Model Hub and 🤗 Transformers, we recommend creating an account.

 

⚠️ Model Hub 및 🤗 Transformers에서 사용할 수 있는 모든 기능을 활용하려면 계정을 만드는 것이 좋습니다.