시애틀에서 100키로미터 쯤 떨어진 곳에 Olympia 라는 곳이 있습니다.

그곳에서 1년동안 은둔 생활을 하면서 틈틈이 AI를 공부했습니다.

그 공부한 내용을 정리하는 의미에서 유투브 채널을 만들 결심을 했습니다.

드디어 첫번째 클립이 업로드 됐어요.

채널 이름은 CatchupAI (https://www.youtube.com/@catchupai).

AI 따라잡기 입니다.

10년전 미국에 와서 IT 기술 따라잡기라는 이름으로 이 티스토리 블로그 활동을 처음 시작했는데요.

이제 AI 따라잡기가 다시 시작 됐습니다.

https://youtu.be/G_X-NtEZ0us?si=heJ5Vz1pa2S9qa7T

작년 중순에 시애틀 지역 한인 IT 엔지니어 모임인 창발이라는 곳에서 AI Study Group에 가입해서 3개월간 Deep Learning Fundamental을 공부 했었어요.

교재는 Dive into Deep Learning 이었구요.

그 내용을 기반으로 Deep Learning Fundamental 을 주제로 PPT를 만들었었습니다.

한두번 이 PPT를 가지고 발표 할 기회가 있었는데요.

그 내용을 유투브로 만들었습니다.

그 중 첫번째 클립이 업로드 된 겁니다.

많은 시청 바라구요. 구독과 좋아요 꼭 부탁 드립니다.

이 클립은 첫번째 Introduce (Target) 과 추가로 인간과 AI 가 이 세상을 인식하는 방법을 비교하는 내용도 있습니다.

AI는 이 세상을 어떻게 이해하고 그것이 인간과 무엇이 다르고 무엇이 같은지 알려 드립니다.

대부분 같지만 약간 다른점이 있는데 그것이 무엇인지 위 유투브 보시고 확인해 보세요.

그 다른점 때문에 생성형 AI의 단점인 Hallucination이 발생하게 되는 겁니다.

그리고 AI 는 어떻게 학습을 하는지 아주 간단한 예제로 AI가 학습하는 과정을 그대로 따라해 봤습니다.

Back propagation이 무엇인지 그리고 그 방법 때문에 ChatGPT 같은 아주 놀라운 성능의 AI 제품이 나올 수 있었지만 동시에 Freshness problem이라는 한계점도 가지고 있다는 점.

위 유투브 내용을 보시면 들어 보지 못했던 흥미로운 점을 이해하실 수 있으실 거예요.

시간과 열정을 가지고 열심히 공부해서 나름대로 이해하고 깨달은 내용을 정리했습니다.

많은 관심과 구독 좋아요 부탁드립니다.

Note: Deep Learning Fundamental 강좌가 끝나면 지금 공부하고 있는 AI Web App 개발과 관련된 내용도 이어질 예정입니다.

제가 지금 개발하고 있는 저만의 AI Web App Portfolio는 아래 웹사이트로 가면 보실 수 있습니다.

https://catchupai.streamlit.app/

https://openai.com/blog/new-embedding-models-and-api-updates

Open AI 에서는 2024년 1월 25일 new generation of embedding models, new GPT-4 Turbo, moderation models, new API usage management tools 를 발표 했습니다.

그리고 조만간 더 저렴한 GPT-3.5 Turbo 모델을 발표할 예정입니다.

이들 새로운 모델에는 다음과 같은 것 들이 포함 됩니다.

• Two new embedding models
• An updated GPT-4 Turbo preview model 
• An updated GPT-3.5 Turbo model
• An updated text moderation model

새로운 임베딩 모델은 더 낮은 가격으로 이용할 수 있습니다.

새로운 임베딩 모델은 아래와 같습니다.

* Small embedding model

- text-embedding-3-small : 고효율 임베딩 모델, 이전 모델인 text-embedding-ada-002 보다 업그레이드 됨

더욱 강력한 성능. text-embedding-ada-002와 text-embedding-3-small을 비교하면 일반적으로 사용되는 다중 언어 검색 벤치마크(MIRACL)의 평균 점수가 31.4%에서 44.0%로 증가했습니다. 영어 과제 벤치마크(MTEB)가 61.0%에서 62.3%로 증가했습니다.

인하 된 가격. text-embedding-3-small은 이전 세대 text-embedding-ada-002 모델보다 훨씬 더 효율적입니다. text-embedding-3-small의 가격은 text-embedding-ada-002에 비해 1,000개 토큰당 가격이 $0.0001에서 $0.00002로 5배 인하되었습니다.

* Large text embedding model

- text-embedding-3-large : 최고 성능 모델. 

text-embedding-3-large는 새로운 차세대 대형 임베딩 모델이며 최대 3072차원의 임베딩을 생성합니다.

더욱 강력한 성능. text-embedding-3-large는 새로운 최고 성능 모델입니다. text-embedding-ada-002와 text-embedding-3-large 비교: MIRACL에서는 평균 점수가 31.4%에서 54.9%로 증가한 반면, MTEB에서는 평균 점수가 61.0%에서 64.6%로 증가했습니다.

text-embedding-3-large의 가격은 $0.00013/1,000토큰입니다.
Embeddings guide 에서 새로운 임베딩 모델 사용에 대해 자세히 알아볼 수 있습니다.

This enables very flexible usage. For example, when using a vector data store that only supports embeddings up to 1024 dimensions long, developers can now still use our best embeddings model text-embedding-3-large and specify a value of 1024 for the dimensions API parameter, which will shorten the embedding down from 3072 dimensions, trading off some accuracy in exchange for the smaller vector size.

이를 통해 매우 유연한 사용이 가능합니다. 예를 들어, 최대 1024차원 길이의 임베딩만 지원하는 벡터 데이터 저장소를 사용할 때 개발자는 이제 최고의 임베딩 모델 text-embedding-3-large를 사용하고 차원 API 매개변수에 값 1024를 지정할 수 있습니다. 3072 차원에서 임베딩을 줄여 벡터 크기가 더 작아지는 대가로 어느 정도 정확도를 희생했습니다.

Next week we are introducing a new GPT-3.5 Turbo model, gpt-3.5-turbo-0125, and for the third time in the past year, we will be decreasing prices on GPT-3.5 Turbo to help our customers scale. Input prices for the new model are reduced by 50% to $0.0005 /1K tokens and output prices are reduced by 25% to $0.0015 /1K tokens. This model will also have various improvements including higher accuracy at responding in requested formats and a fix for a bug which caused a text encoding issue for non-English language function calls.

다음 주에 우리는 새로운 GPT-3.5 Turbo 모델인 gpt-3.5-turbo-0125를 선보일 예정이며, 고객의 확장을 돕기 위해 작년에 세 번째로 GPT-3.5 Turbo의 가격을 인하할 예정입니다. 새 모델의 입력 가격은 $0.0005/1K 토큰으로 50% 인하되고, 출력 가격은 $0.0015/1K 토큰으로 25% 인하됩니다. 또한 이 모델에는 요청된 형식으로 응답할 때 더 높은 정확도와 영어가 아닌 언어 함수 호출에 대한 텍스트 인코딩 문제를 일으키는 버그 수정 등 다양한 개선 사항이 포함됩니다.

Customers using the pinned gpt-3.5-turbo model alias will be automatically upgraded from gpt-3.5-turbo-0613 to gpt-3.5-turbo-0125 two weeks after this model launches.

고정된 gpt-3.5-turbo 모델 별칭을 사용하는 고객은 이 모델 출시 2주 후 gpt-3.5-turbo-0613에서 gpt-3.5-turbo-0125로 자동 업그레이드됩니다.

우리는 앞으로 몇 달 안에 비전이 포함된 GPT-4 Turbo를 일반 출시할 계획입니다.

text-moderation-latest 및 text-moderation-stable 별칭이 이를 가리키도록 업데이트되었습니다. 안전 모범 사례 가이드를 통해 안전한 AI 시스템 구축에 대해 자세히 알아볼 수 있습니다.

둘째, 이제 추적을 활성화한 후 사용량 대시보드 및 사용량 내보내기 기능이 API 키 수준에 지표를 노출합니다. 이를 통해 각각에 대해 별도의 API 키를 보유함으로써 기능, 팀, 제품 또는 프로젝트 수준별 사용량을 간단하게 확인할 수 있습니다.

In the coming months, we plan to further improve the ability for developers to view their API usage and manage API keys, especially in larger organizations.

앞으로 몇 달 안에 우리는 특히 대규모 조직에서 개발자가 자신의 API 사용량을 확인하고 API 키를 관리할 수 있는 기능을 더욱 향상시킬 계획입니다.

For the latest updates on OpenAI's APIs, follow us on X at @OpenAIDevs.

OpenAI API에 대한 최신 업데이트를 보려면 @OpenAIDevs에서 X를 팔로우하세요.

https://openai.com/blog/how-openai-is-approaching-2024-worldwide-elections

Protecting the integrity of elections requires collaboration from every corner of the democratic process, and we want to make sure our technology is not used in a way that could undermine this process. 

선거의 무결성을 보호하려면 민주적 절차의 모든 부분에서 협력이 필요하며, 우리는 우리의 기술이 이 절차를 훼손할 수 있는 방식으로 사용되지 않도록 하고 싶습니다.

Our tools empower people to improve their daily lives and solve complex problems  – from using AI to enhance state services to simplifying medical forms for patients.

우리의 도구는 AI를 사용하여 국가 서비스를 강화하는 것부터 환자를 위한 의료 양식을 단순화하는 것까지 사람들이 일상 생활을 개선하고 복잡한 문제를 해결할 수 있도록 지원합니다.

We want to make sure that our AI systems are built, deployed, and used safely. Like any new technology, these tools come with benefits and challenges. They are also unprecedented, and we will keep evolving our approach as we learn more about how our tools are used.

우리는 AI 시스템이 안전하게 구축, 배포 및 사용되기를 원합니다. 다른 새로운 기술과 마찬가지로 이러한 도구에도 이점과 과제가 있습니다. 이는 또한 전례 없는 일이며, 도구 사용 방법에 대해 더 많이 배우면서 접근 방식을 계속 발전시킬 것입니다.

As we prepare for elections in 2024 across the world’s largest democracies, our approach is to continue our platform safety work by elevating accurate voting information, enforcing measured policies, and improving transparency. We have a cross-functional effort dedicated to election work, bringing together expertise from our safety systems, threat intelligence, legal, engineering, and policy teams to quickly investigate and address potential abuse. 

우리는 세계 최대 민주주의 국가의 2024년 선거를 준비하면서 정확한 투표 정보를 높이고, 신중한 정책을 시행하고, 투명성을 개선하여 플랫폼 안전 작업을 계속하는 것입니다. 우리는 안전 시스템, 위협 인텔리전스, 법률, 엔지니어링 및 정책 팀의 전문 지식을 모아 잠재적인 남용을 신속하게 조사하고 해결하기 위해 선거 업무에 전념하는 다기능적 노력을 기울이고 있습니다.

The following are key initiatives our teams are investing in to prepare for elections this year:

다음은 올해 선거를 준비하기 위해 우리 팀이 투자하고 있는 주요 이니셔티브입니다.

https://openai.com/blog/introducing-chatgpt-team

https://openai.com/blog/introducing-the-gpt-store

Introducing the GPT Store

We’re launching the GPT Store to help you find useful and popular custom versions of ChatGPT.

Explore GPTs

 

 

 

 

It’s been two months since we announced GPTs, and users have already created over 3 million custom versions of ChatGPT. Many builders have shared their GPTs for others to use. Today, we're starting to roll out the GPT Store to ChatGPT Plus, Team and Enterprise users so you can find useful and popular GPTs. Visit chat.openai.com/gpts to explore.

 

GPTs 를 발표한 지 두 달이 지났고 사용자는 이미 300만 개 이상의 ChatGPT 맞춤 버전을 만들었습니다. 많은 빌더가 다른 사람들이 사용할 수 있도록 GPTs 를 공유했습니다. 오늘 우리는 유용하고 인기 있는 GPTs 를 찾을 수 있도록 ChatGPT Plus, Team 및 Enterprise 사용자에게 GPT Store를 출시하기 시작했습니다. chat.openai.com/gpts를 방문하여 살펴보세요.

 

Code Tutor Khan Academy의 Code Tutor로 코딩 기술을 확장하세요

Books로 다음 읽을거리를 찾아보세요

https://openai.com/blog/openai-and-journalism

https://huggingface.co/learn/nlp-course/chapter4/6?fw=pt

End-of-chapter quiz

https://huggingface.co/learn/nlp-course/chapter4/5?fw=pt

Part 1 completed!

This is the end of the first part of the course! Part 2 will be released on November 15th with a big community event, see more information here.

이것으로 강좌의 첫 번째 부분이 끝났습니다! 파트 2는 대규모 커뮤니티 이벤트와 함께 11월 15일에 출시될 예정입니다. 자세한 내용은 여기에서 확인하세요.

You should now be able to fine-tune a pretrained model on a text classification problem (single or pairs of sentences) and upload the result to the Model Hub. To make sure you mastered this first section, you should do exactly that on a problem that interests you (and not necessarily in English if you speak another language)! You can find help in the Hugging Face forums and share your project in this topic once you’re finished.

이제 텍스트 분류 문제(단일 또는 쌍의 문장)에 대해 사전 학습된 모델을 미세 조정하고 결과를 모델 허브에 업로드할 수 있습니다. 이 첫 번째 섹션을 완전히 마스터하려면 관심 있는 문제에 대해 정확하게 해당 섹션을 완료해야 합니다(다른 언어를 사용하는 경우 반드시 영어로 할 필요는 없음)! Hugging Face 포럼에서 도움을 찾을 수 있으며 작업이 끝나면 이 주제에서 프로젝트를 공유할 수 있습니다.

We can’t wait to see what you will build with this!

우리는 당신이 이것으로 무엇을 만들 것인지 기대하고 있습니다!

https://huggingface.co/learn/nlp-course/chapter4/4?fw=pt

Building a model card

The model card is a file which is arguably as important as the model and tokenizer files in a model repository. It is the central definition of the model, ensuring reusability by fellow community members and reproducibility of results, and providing a platform on which other members may build their artifacts.

모델 카드는 모델 저장소의 모델 및 토크나이저 파일만큼 중요한 파일입니다. 이는 모델의 핵심 정의로, 동료 커뮤니티 구성원의 재사용성과 결과 재현성을 보장하고 다른 구성원이 아티팩트를 구축할 수 있는 플랫폼을 제공합니다.

Documenting the training and evaluation process helps others understand what to expect of a model — and providing sufficient information regarding the data that was used and the preprocessing and postprocessing that were done ensures that the limitations, biases, and contexts in which the model is and is not useful can be identified and understood.

교육 및 평가 프로세스를 문서화하면 다른 사람들이 모델에 대해 기대하는 바를 이해하는 데 도움이 됩니다. 사용된 데이터와 수행된 전처리 및 후처리에 대한 충분한 정보를 제공하면 모델이 존재하는 한계, 편향 및 맥락을 보장할 수 있습니다. 유용하지 않은 것을 식별하고 이해할 수 있습니다.

Therefore, creating a model card that clearly defines your model is a very important step. Here, we provide some tips that will help you with this. Creating the model card is done through the README.md file you saw earlier, which is a Markdown file.

따라서 모델을 명확하게 정의하는 모델 카드를 만드는 것은 매우 중요한 단계입니다. 여기서는 이에 도움이 될 몇 가지 팁을 제공합니다. 모델 카드 생성은 앞서 본 Markdown 파일인 README.md 파일을 통해 이루어집니다.

The “model card” concept originates from a research direction from Google, first shared in the paper “Model Cards for Model Reporting” by Margaret Mitchell et al. A lot of information contained here is based on that paper, and we recommend you take a look at it to understand why model cards are so important in a world that values reproducibility, reusability, and fairness.

"모델 카드" 개념은 Margaret Mitchell 외 연구진이 "모델 보고를 위한 모델 카드" 논문에서 처음 공유한 Google의 연구 방향에서 비롯되었습니다. 여기에 포함된 많은 정보는 해당 논문을 기반으로 하며, 재현성, 재사용성 및 공정성을 중시하는 세계에서 모델 카드가 왜 그토록 중요한지 이해하기 위해 이 문서를 살펴보는 것이 좋습니다.

The model card usually starts with a very brief, high-level overview of what the model is for, followed by additional details in the following sections:

모델 카드는 일반적으로 모델의 용도에 대한 매우 간단하고 높은 수준의 개요로 시작하고 다음 섹션의 추가 세부정보가 이어집니다.

• Model description. 모델 설명
• Intended uses & limitations 용도 및 제한 사항
• How to use 사용하는 방법
• Limitations and bias 한계와 편견
• Training data 훈련 데이터
• Training procedure 훈련 절차
• Evaluation results 평가 결과

Let’s take a look at what each of these sections should contain.

각 섹션에 어떤 내용이 포함되어야 하는지 살펴보겠습니다.

Model description

The model description provides basic details about the model. This includes the architecture, version, if it was introduced in a paper, if an original implementation is available, the author, and general information about the model. Any copyright should be attributed here. General information about training procedures, parameters, and important disclaimers can also be mentioned in this section.

모델 설명은 모델에 대한 기본 세부정보를 제공합니다. 여기에는 아키텍처, 버전, 논문에 소개된 경우, 원래 구현이 가능한 경우, 작성자 및 모델에 대한 일반 정보가 포함됩니다. 모든 저작권은 여기에 귀속되어야 합니다. 훈련 절차, 매개변수 및 중요한 면책 조항에 대한 일반 정보도 이 섹션에서 언급될 수 있습니다.

Intended uses & limitations

Here you describe the use cases the model is intended for, including the languages, fields, and domains where it can be applied. This section of the model card can also document areas that are known to be out of scope for the model, or where it is likely to perform suboptimally.

여기서는 모델이 적용될 수 있는 언어, 필드 및 도메인을 포함하여 모델의 사용 사례를 설명합니다. 모델 카드의 이 섹션은 모델의 범위를 벗어나는 것으로 알려진 영역이나 최적이 아닌 성능을 발휘할 가능성이 있는 영역을 문서화할 수도 있습니다.

How to use

This section should include some examples of how to use the model. This can showcase usage of the pipeline() function, usage of the model and tokenizer classes, and any other code you think might be helpful.

이 섹션에는 모델 사용 방법에 대한 몇 가지 예가 포함되어야 합니다. 이는 파이프라인() 함수의 사용법, 모델 및 토크나이저 클래스의 사용법, 도움이 될 수 있다고 생각되는 기타 코드를 보여줄 수 있습니다.

Training data

This part should indicate which dataset(s) the model was trained on. A brief description of the dataset(s) is also welcome.

이 부분은 모델이 훈련된 데이터세트를 나타내야 합니다. 데이터 세트에 대한 간략한 설명도 환영합니다.

Training procedure

In this section you should describe all the relevant aspects of training that are useful from a reproducibility perspective. This includes any preprocessing and postprocessing that were done on the data, as well as details such as the number of epochs the model was trained for, the batch size, the learning rate, and so on.

이 섹션에서는 재현성 관점에서 유용한 훈련의 모든 관련 측면을 설명해야 합니다. 여기에는 데이터에 대해 수행된 전처리 및 후처리뿐만 아니라 모델이 훈련된 시대 수, 배치 크기, 학습 속도 등과 같은 세부 정보도 포함됩니다.

Variable and metrics

Here you should describe the metrics you use for evaluation, and the different factors you are mesuring. Mentioning which metric(s) were used, on which dataset and which dataset split, makes it easy to compare you model’s performance compared to that of other models. These should be informed by the previous sections, such as the intended users and use cases.

여기에서는 평가에 사용하는 측정항목과 측정하는 다양한 요소를 설명해야 합니다. 어떤 측정항목이 사용되었는지, 어떤 데이터세트에 어떤 데이터세트가 분할되었는지 언급하면 모델의 성능을 다른 모델의 성능과 쉽게 비교할 수 있습니다. 이는 의도된 사용자 및 사용 사례와 같은 이전 섹션을 통해 알려야 합니다.

Evaluation results

Finally, provide an indication of how well the model performs on the evaluation dataset. If the model uses a decision threshold, either provide the decision threshold used in the evaluation, or provide details on evaluation at different thresholds for the intended uses.

마지막으로 모델이 평가 데이터 세트에서 얼마나 잘 수행되는지 표시합니다. 모델이 결정 임계값을 사용하는 경우 평가에 사용되는 결정 임계값을 제공하거나 의도된 용도에 대해 다양한 임계값에서의 평가에 대한 세부 정보를 제공합니다.

Example

Check out the following for a few examples of well-crafted model cards:

잘 제작된 모델 카드의 몇 가지 예를 보려면 다음을 확인하세요.

• bert-base-cased
• gpt2
• distilbert

More examples from different organizations and companies are available here.

여기에서 다양한 조직 및 회사의 더 많은 사례를 확인할 수 있습니다.

Note

Model cards are not a requirement when publishing models, and you don’t need to include all of the sections described above when you make one. However, explicit documentation of the model can only benefit future users, so we recommend that you fill in as many of the sections as possible to the best of your knowledge and ability.

모델을 게시할 때 모델 카드는 필수 사항이 아니며, 모델을 만들 때 위에 설명된 모든 섹션을 포함할 필요는 없습니다. 그러나 모델에 대한 명시적인 문서화는 미래의 사용자에게만 도움이 될 수 있으므로 귀하의 지식과 능력을 최대한 활용하여 가능한 한 많은 섹션을 작성하는 것이 좋습니다.

Model card metadata

If you have done a little exploring of the Hugging Face Hub, you should have seen that some models belong to certain categories: you can filter them by tasks, languages, libraries, and more. The categories a model belongs to are identified according to the metadata you add in the model card header.

Hugging Face Hub를 조금 탐색했다면 일부 모델이 특정 범주에 속한다는 것을 알았을 것입니다. 작업, 언어, 라이브러리 등을 기준으로 필터링할 수 있습니다. 모델이 속한 카테고리는 모델 카드 헤더에 추가한 메타데이터에 따라 식별됩니다.

For example, if you take a look at the camembert-base model card, you should see the following lines in the model card header:

예를 들어 camembert-base model card, 를 살펴보면 모델 카드 헤더에 다음 줄이 표시됩니다.

---
language: fr
license: mit
datasets:
- oscar
---

This metadata is parsed by the Hugging Face Hub, which then identifies this model as being a French model, with an MIT license, trained on the Oscar dataset.

이 메타데이터는 Hugging Face Hub에 의해 구문 분석되며, 그런 다음 이 모델을 Oscar 데이터 세트에서 훈련된 MIT 라이선스를 갖춘 프랑스 모델로 식별합니다.

The full model card specification allows specifying languages, licenses, tags, datasets, metrics, as well as the evaluation results the model obtained when training.

The full model card specification allows specifying languages, licenses, tags, datasets, metrics, as well as the evaluation results the model obtained when training.

https://huggingface.co/learn/nlp-course/chapter4/3?fw=pt

Sharing pretrained models

In the steps below, we’ll take a look at the easiest ways to share pretrained models to the 🤗 Hub. There are tools and utilities available that make it simple to share and update models directly on the Hub, which we will explore below.

아래 단계에서는 사전 학습된 모델을 🤗 허브에 공유하는 가장 쉬운 방법을 살펴보겠습니다. 허브에서 직접 모델을 간단하게 공유하고 업데이트할 수 있는 도구와 유틸리티가 있습니다. 이에 대해서는 아래에서 살펴보겠습니다.

https://youtu.be/9yY3RB_GSPM?si=nXnSBdYXIkzb1kna

We encourage all users that train models to contribute by sharing them with the community — sharing models, even when trained on very specific datasets, will help others, saving them time and compute resources and providing access to useful trained artifacts. In turn, you can benefit from the work that others have done!

우리는 모델을 훈련하는 모든 사용자가 커뮤니티와 공유함으로써 기여하도록 권장합니다. 모델을 공유하면 매우 특정한 데이터 세트에 대해 훈련을 받은 경우에도 다른 사람에게 도움이 되고 시간을 절약하며 리소스를 계산하고 유용한 훈련된 아티팩트에 대한 액세스를 제공할 수 있습니다. 그러면 당신도 다른 사람들이 한 일로부터 유익을 얻을 수 있습니다!

There are three ways to go about creating new model repositories:

새 모델 리포지토리를 만드는 방법에는 세 가지가 있습니다.

• Using the push_to_hub API
• push_to_hub API 사용
• Using the huggingface_hub Python library
• Huggingface_hub Python 라이브러리 사용
• Using the web interface
• 웹 인터페이스 사용

Once you’ve created a repository, you can upload files to it via git and git-lfs. We’ll walk you through creating model repositories and uploading files to them in the following sections.

저장소를 생성한 후에는 git 및 git-lfs를 통해 파일을 업로드할 수 있습니다. 다음 섹션에서는 모델 리포지토리를 생성하고 여기에 파일을 업로드하는 과정을 안내합니다.

Using the push_to_hub API

https://youtu.be/Zh0FfmVrKX0?si=afYoVhInYh036Y2T

The simplest way to upload files to the Hub is by leveraging the push_to_hub API.

허브에 파일을 업로드하는 가장 간단한 방법은 push_to_hub API를 활용하는 것입니다.

Before going further, you’ll need to generate an authentication token so that the huggingface_hub API knows who you are and what namespaces you have write access to. Make sure you are in an environment where you have transformers installed (see Setup). If you are in a notebook, you can use the following function to login:

더 진행하기 전에, Huggingface_hub API가 당신 누구인지, 어떤 네임스페이스에 쓰기 액세스 권한이 있는지 알 수 있도록 인증 토큰을 생성해야 합니다. 변압기가 설치된 환경에 있는지 확인하십시오(설정 참조). 노트북에 있는 경우 다음 기능을 사용하여 로그인할 수 있습니다.

from huggingface_hub import notebook_login

notebook_login()

In a terminal, you can run:

huggingface-cli login

In both cases, you should be prompted for your username and password, which are the same ones you use to log in to the Hub. If you do not have a Hub profile yet, you should create one here.

두 경우 모두 허브에 로그인하는 데 사용하는 것과 동일한 사용자 이름과 비밀번호를 입력하라는 메시지가 표시됩니다. 아직 허브 프로필이 없다면 여기에서 생성해야 합니다.

Great! You now have your authentication token stored in your cache folder. Let’s create some repositories!

Great ! 이제 인증 토큰이 캐시 폴더에 저장되었습니다. 저장소를 만들어 봅시다!

If you have played around with the Trainer API to train a model, the easiest way to upload it to the Hub is to set push_to_hub=True when you define your TrainingArguments:

모델을 훈련하기 위해 Trainer API를 사용해 본 경우 이를 허브에 업로드하는 가장 쉬운 방법은 TrainingArguments를 정의할 때 push_to_hub=True를 설정하는 것입니다.

from transformers import TrainingArguments

training_args = TrainingArguments(
    "bert-finetuned-mrpc", save_strategy="epoch", push_to_hub=True
)

When you call trainer.train(), the Trainer will then upload your model to the Hub each time it is saved (here every epoch) in a repository in your namespace. That repository will be named like the output directory you picked (here bert-finetuned-mrpc) but you can choose a different name with hub_model_id = "a_different_name".

Trainer.train()을 호출하면 Trainer는 모델이 네임스페이스의 저장소에 저장될 때마다(여기에서는 매 에포크마다) 모델을 허브에 업로드합니다. 해당 저장소의 이름은 선택한 출력 디렉터리(여기서는 bert-finetuned-mrpc)와 같이 지정되지만,hub_model_id = "a_ Different_name"을 사용하여 다른 이름을 선택할 수 있습니다.

To upload your model to an organization you are a member of, just pass it with hub_model_id = "my_organization/my_repo_name".

당신이 소속된 organization  에 모델을 업로드하려면, Hub_model_id = "my_organization/my_repo_name"으로 전달하세요.

Once your training is finished, you should do a final trainer.push_to_hub() to upload the last version of your model. It will also generate a model card with all the relevant metadata, reporting the hyperparameters used and the evaluation results! Here is an example of the content you might find in a such a model card:

훈련이 완료되면 최종 training.push_to_hub()를 수행하여 모델의 마지막 버전을 업로드해야 합니다. 또한 사용된 하이퍼파라미터와 평가 결과를 보고하는 모든 관련 메타데이터가 포함된 모델 카드를 생성합니다! 다음은 이러한 모델 카드에서 찾을 수 있는 콘텐츠의 예입니다.

At a lower level, accessing the Model Hub can be done directly on models, tokenizers, and configuration objects via their push_to_hub() method. This method takes care of both the repository creation and pushing the model and tokenizer files directly to the repository. No manual handling is required, unlike with the API we’ll see below.

낮은 수준에서는 모델 허브에 대한 액세스가 push_to_hub() 메서드를 통해 모델, 토크나이저 및 구성 개체에서 직접 수행될 수 있습니다. 이 방법은 저장소 생성과 모델 및 토크나이저 파일을 저장소에 직접 푸시하는 작업을 모두 처리합니다. 아래에서 볼 API와 달리 수동 처리가 필요하지 않습니다.

To get an idea of how it works, let’s first initialize a model and a tokenizer:

작동 방식에 대한 아이디어를 얻으려면 먼저 모델과 토크나이저를 초기화해 보겠습니다.

from transformers import AutoModelForMaskedLM, AutoTokenizer

checkpoint = "camembert-base"

model = AutoModelForMaskedLM.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

You’re free to do whatever you want with these — add tokens to the tokenizer, train the model, fine-tune it. Once you’re happy with the resulting model, weights, and tokenizer, you can leverage the push_to_hub() method directly available on the model object:

토크나이저에 토큰을 추가하고, 모델을 훈련하고, 미세 조정하는 등 원하는 대로 무엇이든 자유롭게 할 수 있습니다. 결과 모델, 가중치 및 토크나이저에 만족하면 모델 객체에서 직접 사용할 수 있는 push_to_hub() 메서드를 활용할 수 있습니다.

model.push_to_hub("dummy-model")

This will create the new repository dummy-model in your profile, and populate it with your model files. Do the same with the tokenizer, so that all the files are now available in this repository:

그러면 프로필에 새 저장소 더미 모델이 생성되고 모델 파일이 채워집니다. 이제 토크나이저에서도 동일한 작업을 수행하여 이제 이 저장소에서 모든 파일을 사용할 수 있습니다.

tokenizer.push_to_hub("dummy-model")

If you belong to an organization, simply specify the organization argument to upload to that organization’s namespace:

조직에 속해 있는 경우 해당 조직의 네임스페이스에 업로드할 조직 인수를 지정하면 됩니다.

tokenizer.push_to_hub("dummy-model", organization="huggingface")

Now head to the Model Hub to find your newly uploaded model: https://huggingface.co/user-or-organization/dummy-model.

이제 모델 허브로 이동하여 새로 업로드된 모델을 찾으세요: https://huggingface.co/user-or-organization/dummy-model.

Click on the “Files and versions” tab, and you should see the files visible in the following screenshot:

"파일 및 버전" 탭을 클릭하면 다음 스크린샷에 표시된 파일을 볼 수 있습니다.

✏️ Try it out! Take the model and tokenizer associated with the bert-base-cased checkpoint and upload them to a repo in your namespace using the push_to_hub() method. Double-check that the repo appears properly on your page before deleting it.

✏️ 한번 사용해 보세요! bert-base 케이스 체크포인트와 연관된 모델 및 토크나이저를 가져와서 push_to_hub() 메소드를 사용하여 네임스페이스의 저장소에 업로드하십시오. 저장소를 삭제하기 전에 페이지에 저장소가 제대로 나타나는지 다시 확인하세요.

As you’ve seen, the push_to_hub() method accepts several arguments, making it possible to upload to a specific repository or organization namespace, or to use a different API token. We recommend you take a look at the method specification available directly in the 🤗 Transformers documentation to get an idea of what is possible.

본 것처럼 push_to_hub() 메서드는 여러 인수를 허용하므로 특정 저장소나 조직 네임스페이스에 업로드하거나 다른 API 토큰을 사용할 수 있습니다. 무엇이 가능한지에 대한 아이디어를 얻으려면 🤗 Transformers 문서에서 직접 제공되는 메서드 사양을 살펴보는 것이 좋습니다.

The push_to_hub() method is backed by the huggingface_hub Python package, which offers a direct API to the Hugging Face Hub. It’s integrated within 🤗 Transformers and several other machine learning libraries, like allenlp. Although we focus on the 🤗 Transformers integration in this chapter, integrating it into your own code or library is simple.

push_to_hub() 메서드는 Hugging Face Hub에 직접 API를 제공하는 Huggingface_hub Python 패키지의 지원을 받습니다. 이는 🤗 Transformers 및 AllenLP와 같은 기타 여러 기계 학습 라이브러리 내에 통합되어 있습니다. 이 장에서는 🤗 Transformers 통합에 중점을 두지만 이를 자신의 코드나 라이브러리에 통합하는 것은 간단합니다.

Jump to the last section to see how to upload files to your newly created repository!

새로 생성된 저장소에 파일을 업로드하는 방법을 보려면 마지막 섹션으로 이동하세요!

Using the huggingface_hub Python library

The huggingface_hub Python library is a package which offers a set of tools for the model and datasets hubs. It provides simple methods and classes for common tasks like getting information about repositories on the hub and managing them. It provides simple APIs that work on top of git to manage those repositories’ content and to integrate the Hub in your projects and libraries.

Huggingface_hub Python 라이브러리는 모델 및 데이터 세트 허브를 위한 도구 세트를 제공하는 패키지입니다. 허브의 리포지토리에 대한 정보 가져오기 및 관리와 같은 일반적인 작업을 위한 간단한 메서드와 클래스를 제공합니다. git 위에서 작동하여 해당 리포지토리의 콘텐츠를 관리하고 프로젝트 및 라이브러리에 허브를 통합하는 간단한 API를 제공합니다.

Similarly to using the push_to_hub API, this will require you to have your API token saved in your cache. In order to do this, you will need to use the login command from the CLI, as mentioned in the previous section (again, make sure to prepend these commands with the ! character if running in Google Colab):

push_to_hub API를 사용하는 것과 마찬가지로 API 토큰을 캐시에 저장해야 합니다. 이렇게 하려면 이전 섹션에서 언급한 대로 CLI에서 로그인 명령을 사용해야 합니다(Google Colab에서 실행하는 경우 이 명령 앞에 ! 문자를 추가해야 합니다).

huggingface-cli login

The huggingface_hub package offers several methods and classes which are useful for our purpose. Firstly, there are a few methods to manage repository creation, deletion, and others:

Huggingface_hub 패키지는 우리의 목적에 유용한 여러 메서드와 클래스를 제공합니다. 첫째, 저장소 생성, 삭제 등을 관리하는 몇 가지 방법이 있습니다.

from huggingface_hub import (
    # User management
    login,
    logout,
    whoami,

    # Repository creation and management
    create_repo,
    delete_repo,
    update_repo_visibility,

    # And some methods to retrieve/change information about the content
    list_models,
    list_datasets,
    list_metrics,
    list_repo_files,
    upload_file,
    delete_file,
)

Additionally, it offers the very powerful Repository class to manage a local repository. We will explore these methods and that class in the next few section to understand how to leverage them.

또한 로컬 저장소를 관리할 수 있는 매우 강력한 Repository 클래스를 제공합니다. 다음 몇 섹션에서 이러한 메서드와 해당 클래스를 살펴보고 이를 활용하는 방법을 이해할 것입니다.

The create_repo method can be used to create a new repository on the hub:

create_repo 메소드를 사용하여 허브에 새 저장소를 생성할 수 있습니다.

from huggingface_hub import create_repo

create_repo("dummy-model")

This will create the repository dummy-model in your namespace. If you like, you can specify which organization the repository should belong to using the organization argument:

그러면 네임스페이스에 저장소 더미 모델이 생성됩니다. 원하는 경우 조직 인수를 사용하여 저장소가 속해야 하는 조직을 지정할 수 있습니다.

from huggingface_hub import create_repo

create_repo("dummy-model", organization="huggingface")

This will create the dummy-model repository in the huggingface namespace, assuming you belong to that organization. Other arguments which may be useful are:

그러면 귀하가 해당 조직에 속해 있다는 가정 하에 Huggingface 네임스페이스에 더미 모델 저장소가 생성됩니다. 유용할 수 있는 다른 인수는 다음과 같습니다.

• private, in order to specify if the repository should be visible from others or not.
• 비공개, 저장소가 다른 사람에게 표시되어야 하는지 여부를 지정합니다.
• token, if you would like to override the token stored in your cache by a given token.
• 토큰: 특정 토큰으로 캐시에 저장된 토큰을 재정의하려는 경우.
• repo_type, if you would like to create a dataset or a space instead of a model. Accepted values are "dataset" and "space".
• repo_type, 모델 대신 데이터 세트나 공간을 생성하려는 경우. 허용되는 값은 "dataset" 및 "space"입니다.

Once the repository is created, we should add files to it! Jump to the next section to see the three ways this can be handled.

저장소가 생성되면 여기에 파일을 추가해야 합니다! 이 문제를 처리할 수 있는 세 가지 방법을 보려면 다음 섹션으로 이동하세요.

Using the web interface

The web interface offers tools to manage repositories directly in the Hub. Using the interface, you can easily create repositories, add files (even large ones!), explore models, visualize diffs, and much more.

To create a new repository, visit huggingface.co/new:

웹 인터페이스는 허브에서 직접 저장소를 관리할 수 있는 도구를 제공합니다. 인터페이스를 사용하면 저장소 생성, 파일 추가(큰 파일도 포함), 모델 탐색, 차이점 시각화 등을 쉽게 수행할 수 있습니다.

새 저장소를 만들려면 Huggingface.co/new를 방문하세요.

First, specify the owner of the repository: this can be either you or any of the organizations you’re affiliated with. If you choose an organization, the model will be featured on the organization’s page and every member of the organization will have the ability to contribute to the repository.

먼저 저장소의 소유자를 지정하십시오. 이는 귀하 또는 귀하가 소속된 조직일 수 있습니다. 조직을 선택하면 해당 모델이 해당 조직의 페이지에 표시되며 조직의 모든 구성원은 저장소에 기여할 수 있습니다.

Next, enter your model’s name. This will also be the name of the repository. Finally, you can specify whether you want your model to be public or private. Private models are hidden from public view.

다음으로 모델 이름을 입력하세요. 이는 저장소의 이름이기도 합니다. 마지막으로 모델을 공개할지 비공개할지 지정할 수 있습니다. 비공개 모델은 공개적으로 표시되지 않습니다.

After creating your model repository, you should see a page like this:

모델 리포지토리를 생성하면 다음과 같은 페이지가 표시됩니다.

This is where your model will be hosted. To start populating it, you can add a README file directly from the web interface.

이곳이 귀하의 모델이 호스팅되는 곳입니다. 채우기를 시작하려면 웹 인터페이스에서 직접 README 파일을 추가하면 됩니다.

The README file is in Markdown — feel free to go wild with it! The third part of this chapter is dedicated to building a model card. These are of prime importance in bringing value to your model, as they’re where you tell others what it can do.

README 파일은 Markdown에 있습니다. 마음껏 사용해 보세요! 이 장의 세 번째 부분은 모델 카드를 만드는 데 전념합니다. 이는 모델이 무엇을 할 수 있는지 다른 사람에게 알려주기 때문에 모델에 가치를 부여하는 데 가장 중요합니다.

If you look at the “Files and versions” tab, you’ll see that there aren’t many files there yet — just the README.md you just created and the .gitattributes file that keeps track of large files.

"파일 및 버전" 탭을 보면 아직 파일이 많지 않다는 것을 알 수 있습니다. 방금 생성한 README.md와 대용량 파일을 추적하는 .gitattributes 파일만 있습니다.

We’ll take a look at how to add some new files next.

다음에는 새 파일을 추가하는 방법을 살펴보겠습니다.

Uploading the model files

The system to manage files on the Hugging Face Hub is based on git for regular files, and git-lfs (which stands for Git Large File Storage) for larger files.

Hugging Face Hub의 파일 관리 시스템은 일반 파일은 git, 대용량 파일은 git-lfs(Git Large File Storage의 약자)를 기반으로 합니다.

In the next section, we go over three different ways of uploading files to the Hub: through huggingface_hub and through git commands.

다음 섹션에서는 Huggingface_hub 및 git 명령을 통해 허브에 파일을 업로드하는 세 가지 방법을 살펴보겠습니다.

The upload_file approach

Using upload_file does not require git and git-lfs to be installed on your system. It pushes files directly to the 🤗 Hub using HTTP POST requests. A limitation of this approach is that it doesn’t handle files that are larger than 5GB in size. If your files are larger than 5GB, please follow the two other methods detailed below.

upload_file을 사용하면 시스템에 git 및 git-lfs를 설치할 필요가 없습니다. HTTP POST 요청을 사용하여 파일을 🤗 허브에 직접 푸시합니다. 이 접근 방식의 한계는 크기가 5GB보다 큰 파일을 처리하지 못한다는 것입니다. 파일이 5GB보다 큰 경우 아래에 설명된 두 가지 다른 방법을 따르세요.

The API may be used as follows:

API는 다음과 같이 사용될 수 있습니다:

from huggingface_hub import upload_file

upload_file(
    "<path_to_file>/config.json",
    path_in_repo="config.json",
    repo_id="<namespace>/dummy-model",
)

This will upload the file config.json available at <path_to_file> to the root of the repository as config.json, to the dummy-model repository. Other arguments which may be useful are:

그러면 <path_to_file>에서 사용 가능한 config.json 파일이 저장소 루트에 config.json으로 더미 모델 저장소에 업로드됩니다. 유용할 수 있는 다른 인수는 다음과 같습니다.

• token, if you would like to override the token stored in your cache by a given token.
• 토큰: 특정 토큰으로 캐시에 저장된 토큰을 재정의하려는 경우.
• repo_type, if you would like to upload to a dataset or a space instead of a model. Accepted values are "dataset" and "space".
• repo_type, 모델 대신 데이터세트나 공간에 업로드하려는 경우. 허용되는 값은 "dataset" 및 "space"입니다.

The Repository class

The Repository class manages a local repository in a git-like manner. It abstracts most of the pain points one may have with git to provide all features that we require.

Repository 클래스는 Git과 같은 방식으로 로컬 저장소를 관리합니다. 이는 필요한 모든 기능을 제공하기 위해 git에서 발생할 수 있는 대부분의 문제점을 추상화합니다.

Using this class requires having git and git-lfs installed, so make sure you have git-lfs installed (see here for installation instructions) and set up before you begin.

이 클래스를 사용하려면 git 및 git-lfs가 설치되어 있어야 하므로 시작하기 전에 git-lfs가 설치되어 있는지 확인하고(설치 지침은 여기 참조) 설정하세요.

In order to start playing around with the repository we have just created, we can start by initialising it into a local folder by cloning the remote repository:

방금 생성한 저장소를 가지고 놀기 위해 원격 저장소를 복제하여 로컬 폴더로 초기화하는 것부터 시작할 수 있습니다.

from huggingface_hub import Repository

repo = Repository("<path_to_dummy_folder>", clone_from="<namespace>/dummy-model")

This created the folder <path_to_dummy_folder> in our working directory. This folder only contains the .gitattributes file as that’s the only file created when instantiating the repository through create_repo.

그러면 작업 디렉터리에 <path_to_dummy_folder> 폴더가 생성되었습니다. 이 폴더에는 create_repo를 통해 저장소를 인스턴스화할 때 생성되는 유일한 파일인 .gitattributes 파일만 포함됩니다.

From this point on, we may leverage several of the traditional git methods:

이 시점부터 우리는 전통적인 git 메소드 중 몇 가지를 활용할 수 있습니다:

repo.git_pull()
repo.git_add()
repo.git_commit()
repo.git_push()
repo.git_tag()

And others! We recommend taking a look at the Repository documentation available here for an overview of all available methods.

다른 사람! 사용 가능한 모든 방법에 대한 개요를 보려면 여기에서 제공되는 리포지토리 문서를 살펴보는 것이 좋습니다.

At present, we have a model and a tokenizer that we would like to push to the hub. We have successfully cloned the repository, we can therefore save the files within that repository.

현재 우리는 허브에 푸시하려는 모델과 토크나이저를 보유하고 있습니다. 저장소를 성공적으로 복제했으므로 해당 저장소 내에 파일을 저장할 수 있습니다.

We first make sure that our local clone is up to date by pulling the latest changes:

먼저 최신 변경 사항을 가져와서 로컬 클론이 최신 상태인지 확인합니다.

repo.git_pull()

Once that is done, we save the model and tokenizer files:

완료되면 모델과 토크나이저 파일을 저장합니다.

model.save_pretrained("<path_to_dummy_folder>")
tokenizer.save_pretrained("<path_to_dummy_folder>")

The <path_to_dummy_folder> now contains all the model and tokenizer files. We follow the usual git workflow by adding files to the staging area, committing them and pushing them to the hub:

이제 <path_to_dummy_folder>에는 모든 모델 및 토크나이저 파일이 포함됩니다. 우리는 스테이징 영역에 파일을 추가하고 커밋한 후 허브에 푸시하는 방식으로 일반적인 git 워크플로를 따릅니다.

repo.git_add()
repo.git_commit("Add model and tokenizer files")
repo.git_push()

Congratulations! You just pushed your first files on the hub.

축하해요! 방금 첫 번째 파일을 허브에 푸시했습니다.

The git-based approach

This is the very barebones approach to uploading files: we’ll do so with git and git-lfs directly. Most of the difficulty is abstracted away by previous approaches, but there are a few caveats with the following method so we’ll follow a more complex use-case.

이것은 파일 업로드에 대한 매우 기본적인 접근 방식입니다. git 및 git-lfs를 직접 사용하여 수행합니다. 대부분의 어려움은 이전 접근 방식으로 추상화되지만 다음 방법에는 몇 가지 주의 사항이 있으므로 보다 복잡한 사용 사례를 따르겠습니다.

Using this class requires having git and git-lfs installed, so make sure you have git-lfs installed (see here for installation instructions) and set up before you begin.

이 클래스를 사용하려면 git 및 git-lfs가 설치되어 있어야 하므로 시작하기 전에 git-lfs가 설치되어 있는지 확인하고(설치 지침은 여기 참조) 설정하세요.

First start by initializing git-lfs: 먼저 git-lfs를 초기화하여 시작하세요:

git lfs install

Updated git hooks.
Git LFS initialized.

Once that’s done, the first step is to clone your model repository:

완료되면 첫 번째 단계는 모델 저장소를 복제하는 것입니다.

git clone https://huggingface.co/<namespace>/<your-model-id>

My username is lysandre and I’ve used the model name dummy, so for me the command ends up looking like the following:

내 사용자 이름은 lysandre이고 모델 이름은 dummy를 사용했기 때문에 명령은 다음과 같이 표시됩니다.

git clone https://huggingface.co/lysandre/dummy

I now have a folder named dummy in my working directory. I can cd into the folder and have a look at the contents:

이제 작업 디렉터리에 dummy라는 폴더가 생겼습니다. 폴더에 CD를 넣고 내용을 볼 수 있습니다.

cd dummy && ls

README.md

If you just created your repository using Hugging Face Hub’s create_repo method, this folder should only contain a hidden .gitattributes file. If you followed the instructions in the previous section to create a repository using the web interface, the folder should contain a single README.md file alongside the hidden .gitattributes file, as shown here.

Hugging Face Hub의 create_repo 메소드를 사용하여 저장소를 방금 생성한 경우 이 폴더에는 숨겨진 .gitattributes 파일만 포함되어야 합니다. 이전 섹션의 지침에 따라 웹 인터페이스를 사용하여 저장소를 만든 경우 폴더에는 여기에 표시된 것처럼 숨겨진 .gitattributes 파일과 함께 단일 README.md 파일이 포함되어야 합니다.

Adding a regular-sized file, such as a configuration file, a vocabulary file, or basically any file under a few megabytes, is done exactly as one would do it in any git-based system. However, bigger files must be registered through git-lfs in order to push them to huggingface.co.

구성 파일, 어휘 파일 또는 기본적으로 몇 메가바이트 미만의 파일과 같은 일반 크기의 파일을 추가하는 것은 모든 git 기반 시스템에서 수행하는 것과 똑같이 수행됩니다. 그러나 더 큰 파일을 Huggingface.co로 푸시하려면 git-lfs를 통해 등록해야 합니다.

Let’s go back to Python for a bit to generate a model and tokenizer that we’d like to commit to our dummy repository:

Python으로 돌아가서 더미 저장소에 커밋할 모델과 토크나이저를 생성해 보겠습니다.

from transformers import AutoModelForMaskedLM, AutoTokenizer

checkpoint = "camembert-base"

model = AutoModelForMaskedLM.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)

# Do whatever with the model, train it, fine-tune it...

model.save_pretrained("<path_to_dummy_folder>")
tokenizer.save_pretrained("<path_to_dummy_folder>")

Now that we’ve saved some model and tokenizer artifacts, let’s take another look at the dummy folder:

이제 일부 모델과 토크나이저 아티팩트를 저장했으므로 더미 폴더를 다시 살펴보겠습니다.

ls

config.json  pytorch_model.bin  README.md  sentencepiece.bpe.model  special_tokens_map.json tokenizer_config.json  tokenizer.json

If you look at the file sizes (for example, with ls -lh), you should see that the model state dict file (pytorch_model.bin) is the only outlier, at more than 400 MB.

파일 크기(예: ls -lh 사용)를 살펴보면 모델 상태 dict 파일(pytorch_model.bin)이 400MB를 초과하는 유일한 이상값임을 알 수 있습니다.

We can now go ahead and proceed like we would usually do with traditional Git repositories. We can add all the files to Git’s staging environment using the git add command:

이제 기존 Git 리포지토리에서 일반적으로 하던 것처럼 진행할 수 있습니다. git add 명령을 사용하여 Git의 스테이징 환경에 모든 파일을 추가할 수 있습니다.

git add .

We can then have a look at the files that are currently staged:

그런 다음 현재 준비된 파일을 살펴볼 수 있습니다.

git status

On branch main
Your branch is up to date with 'origin/main'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
  modified:   .gitattributes
	new file:   config.json
	new file:   pytorch_model.bin
	new file:   sentencepiece.bpe.model
	new file:   special_tokens_map.json
	new file:   tokenizer.json
	new file:   tokenizer_config.json

Similarly, we can make sure that git-lfs is tracking the correct files by using its status command:

마찬가지로, status 명령을 사용하여 git-lfs가 올바른 파일을 추적하고 있는지 확인할 수 있습니다.

git lfs status

On branch main
Objects to be pushed to origin/main:


Objects to be committed:

	config.json (Git: bc20ff2)
	pytorch_model.bin (LFS: 35686c2)
	sentencepiece.bpe.model (LFS: 988bc5a)
	special_tokens_map.json (Git: cb23931)
	tokenizer.json (Git: 851ff3e)
	tokenizer_config.json (Git: f0f7783)

Objects not staged for commit:

We can see that all files have Git as a handler, except pytorch_model.bin and sentencepiece.bpe.model, which have LFS. Great!

LFS가 있는 pytorch_model.bin 및 문장 조각.bpe.model을 제외하고 모든 파일에는 핸들러로 Git이 있는 것을 볼 수 있습니다. Great !

Let’s proceed to the final steps, committing and pushing to the huggingface.co remote repository:

Huggingface.co 원격 저장소에 커밋하고 푸시하는 마지막 단계를 진행해 보겠습니다.

git commit -m "First model version"

[main b08aab1] First model version
 7 files changed, 29027 insertions(+)
  6 files changed, 36 insertions(+)
 create mode 100644 config.json
 create mode 100644 pytorch_model.bin
 create mode 100644 sentencepiece.bpe.model
 create mode 100644 special_tokens_map.json
 create mode 100644 tokenizer.json
 create mode 100644 tokenizer_config.json

If we take a look at the model repository when this is finished, we can see all the recently added files:

이 작업이 완료된 후 모델 저장소를 살펴보면 최근에 추가된 모든 파일을 볼 수 있습니다.

The UI allows you to explore the model files and commits and to see the diff introduced by each commit:

UI를 사용하면 모델 파일과 커밋을 탐색하고 각 커밋에서 도입된 차이점을 확인할 수 있습니다.