끄적끄적

테디님이 올려놓으신 유튜브 영상을 보고 Macbook M1에서 따라해봤습니다.  중간에 몇가지 설정에 맞게 약간 변경해가면서 테스트 합니다.

https://www.youtube.com/watch?v=VkcaigvTrug&t=23s

# GPU 모니터링
% sudo asitop

# LangServe 실행
% python server.py

# ngrok으로 external 서비스
% ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

# PDF를 RAG
% streamlit run main.py

conda 가상환경 만들고 requirements.txt로 필요한 모듈 설치

conda create -n llm311 python=3.11

% conda env list
# conda environments:
#
base                  *  /Users/dongsik/miniconda
llm                      /Users/dongsik/miniconda/envs/llm
llm311                   /Users/dongsik/miniconda/envs/llm311

% conda activate llm311

% python -V
Python 3.11.9

% pip list
Package    Version
---------- -------
pip        23.3.1
setuptools 68.2.2
wheel      0.41.2

예제 github을 내 github으로 fork 한후 내 PC에 clone 받아서 내환경에 맞게 수정하면서 진행합니다.

teddy github : https://github.com/teddylee777/langserve_ollama

내 github : https://github.com/dongshik/langserve_ollama

% ll
total 1000
drwxr-xr-x@ 12 dongsik  staff     384 Apr 20 09:22 .
drwxr-xr-x   4 dongsik  staff     128 Apr 19 16:35 ..
drwxr-xr-x@ 14 dongsik  staff     448 Apr 19 16:40 .git
-rw-r--r--@  1 dongsik  staff      50 Apr 19 16:35 .gitignore
-rw-r--r--@  1 dongsik  staff    3343 Apr 19 16:35 README.md
drwxr-xr-x@  8 dongsik  staff     256 Apr 19 16:35 app
drwxr-xr-x@  8 dongsik  staff     256 Apr 19 16:35 example
drwxr-xr-x@  3 dongsik  staff      96 Apr 19 16:35 images
drwxr-xr-x@  4 dongsik  staff     128 Apr 19 16:35 ollama-modelfile
-rw-r--r--@  1 dongsik  staff  481043 Apr 19 16:35 poetry.lock
-rw-r--r--@  1 dongsik  staff     659 Apr 19 16:35 pyproject.toml
-rw-r--r--@  1 dongsik  staff   14983 Apr 19 16:35 requirements.txt

pip install -r requirements.txt

% pip install -r requirements.txt
Ignoring colorama: markers 'python_version >= "3.11.dev0" and python_version < "3.12.dev0" and platform_system == "Windows"' don't match your environment

% pip list | grep lang
langchain                  0.1.16
langchain-community        0.0.32
langchain-core             0.1.42
langchain-openai           0.1.3
langchain-text-splitters   0.0.1
langchainhub               0.1.15
langdetect                 1.0.9
langserve                  0.0.51
langsmith                  0.1.47

% pip list | grep huggingface
huggingface-hub            0.22.2

Huggingface에서 모델 Download 받고 Ollama에 EEVE Q5 모델 등록하고 구동

huggingface-cli download \
  heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF \
  ggml-model-Q5_K_M.gguf \
  --local-dir /Users/dongsik/GitHub/teddylee777/langserve_ollama/ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0 \
  --local-dir-use-symlinks False
  
Consider using `hf_transfer` for faster downloads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.
downloading https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/resolve/main/ggml-model-Q5_K_M.gguf to /Users/dongsik/.cache/huggingface/hub/tmpkuuur4ki
ggml-model-Q5_K_M.gguf:  37%|███████████████████████████▌                                               | 2.81G/7.65G [04:35<09:55, 8.13MB/s]

% ll -sh
total 14954512
       0 drwxr-xr-x@ 5 dongsik  staff   160B Apr 20 10:02 .
       0 drwxr-xr-x@ 4 dongsik  staff   128B Apr 20 10:02 ..
       8 -rw-r--r--@ 1 dongsik  staff   369B Apr 19 16:35 Modelfile
       8 -rw-r--r--@ 1 dongsik  staff   419B Apr 19 16:35 Modelfile-V02
14954496 -rw-r--r--  1 dongsik  staff   7.1G Apr 20 10:02 ggml-model-Q5_K_M.gguf

<경로>/langserve_ollama/ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

FROM ggml-model-Q5_K_M.gguf

TEMPLATE """{{- if .System }}
<s>{{ .System }}</s>
{{- end }}
<s>Human:
{{ .Prompt }}</s>
<s>Assistant:
"""

SYSTEM """A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."""

PARAMETER TEMPERATURE 0
PARAMETER stop <s>
PARAMETER stop </s>

모델파일 설정하지 않으면 답변이 끝났을때 이상하게 대답할수도있기때문에 필요합니다. 

System prompt가 있다면 중간(.System) 위치에 넣어으라는 의미이며 여기서는 'SYSTEM'이 이자리를 치환하게 됩니다.

그다음 <s> 스페셜 토큰이 앞에 붙어서 사용자 즉 Human의 질문 .Prompt가 들어가게 됩니다.

그후 모델 Assistant가 받아서 답변하게 됩니다. 

※ Note!!

Modelfile에서 <s>는 문장의 시작을 나타내는 특수 토큰입니다. 이것은 "문장의 시작"을 나타내기 위해 사용됩니다. 예를 들어, 자연어 처리 작업에서 모델이 문장의 시작을 식별하고, 이에 따라 적절한 처리를 수행할 수 있도록 합니다. 이것은 토큰화된 데이터의 일부로서 모델에 제공됩니다.

tokenizer.chat_template

https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0

Prompt Template

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: {prompt}
Assistant:

ollama 목록 확인 

% ollama list
NAME    	ID          	SIZE  	MODIFIED
eeve:q4 	68f4c2c2d9fe	6.5 GB	7 days ago
gemma:2b	b50d6c999e59	1.7 GB	10 days ago

ollama가 잘 구동되어 있는지 확인합니다.

% ps -ef | grep ollama
  501  3715  3691   0 Wed01PM ??         0:29.51 /Applications/Ollama.app/Contents/Resources/ollama serve
  501  4430     1   0 Wed01PM ??         0:00.03 /Applications/Ollama.app/Contents/Frameworks/Squirrel.framework/Resources/ShipIt com.electron.ollama.ShipIt /Users/dongsik/Library/Caches/com.electron.ollama.ShipIt/ShipItState.plist
  501 61608  3197   0 10:45AM ttys002    0:00.01 grep ollama

새로받은 모델을 ollama에 등록합니다

ollama create eeve:q5 -f ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

저는 위의 Modelfile로 ollama등록할려고 하니 "Error: unknown parameter 'TEMPERATURE'"가 발생했습니다. 
그래서 소문자 temperature로 변경해서 생성되었습니다.
만일 동일한 에러가 발생한다면 소문자 temperature로 변경해서 생성해보시기 바랍니다. 

% ollama create eeve:q5 -f ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

transferring model data
creating model layer
creating template layer
creating system layer
creating parameters layer
creating config layer
using already created layer sha256:b9e3d1ad5e8aa6db09610d4051820f06a5257b7d7f0b06c00630e376abcfa4c1
writing layer sha256:6b70a2ad0d545ca50d11b293ba6f6355eff16363425c8b163289014cf19311fc
writing layer sha256:1fa69e2371b762d1882b0bd98d284f312a36c27add732016e12e52586f98a9f5
writing layer sha256:3ab8c1bbd3cd85e1b39b09f5ff9a76e64da20ef81c22ec0937cc2e7076f1a81c
writing layer sha256:d86595b443c06710a3e5ba27700c6a93ded80100ff1aa808a7f3444ff529fa70
writing manifest
success

% ollama list
NAME    	ID          	SIZE  	MODIFIED
eeve:q4 	68f4c2c2d9fe	6.5 GB	7 days ago
eeve:q5 	0732d4a47219	7.7 GB	7 minutes ago
gemma:2b	b50d6c999e59	1.7 GB	10 days ago

ollama run eeve:q5

% ollama run eeve:q5
>>> 대한민국의 수도는 어디야?
안녕하세요! 대한민국의 수도에 대해 궁금해하시는군요. 서울이 바로 그 곳입니다! 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를 가진 대도시로,
세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며, 많은 바와 클럽
관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?

>>>

아래 문구로 질문해보겠습니다.

한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주

>>> 한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주
대한민국의 수도에 대한 질문에 답변해 주셔서 감사합니다! 정답은 (D) 서울입니다. 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를
대도시로, 세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며,
많은 바와 클럽이 관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?

질문과 동시에 답변이 나오는것처럼 작동합니다. 속도도 좋고 답변의 퀄리티도 좋습니다.

>>> 다음 지문을 읽고 문제에 답하시오.
...
... ---
...
... 1950년 7월, 한국 전쟁 초기에 이승만 대통령은 맥아더 장군에게 유격대원들을 북한군의 후방에 침투시키는 방안을 제안했다. 이후, 육군본부는 육본직할 유격대와 육본 독립 유격대를 편성했다. 국군은 포항과 인접한 장사동 지역에 상륙작
... 전을 수행할 부대로 독립 제1유격대대를 선정했다. 육군본부는 독립 제1유격대대에 동해안의 장사동 해안에 상륙작전을 감행하여 북한군 제2군단의 보급로를 차단하고 국군 제1군단의 작전을 유리하게 하기 위한 작전명령(육본 작명 제174호)
... 을 하달했다. 9월 14일, 독립 제1유격대대는 부산에서 LST 문산호에 승선하여 영덕군의 장사동으로 출항했다.
...
... 1950년 9월 15일, 독립 제1유격대대는 장사동 해안에 상륙을 시도하였으나 태풍 케지아로 인한 높은 파도와 안개로 인해 어려움을 겪었다. LST 문산호는 북한군의 사격과 파도로 인해 좌초되었고, 상륙부대는 09:00시경에 전원이
... 상륙을 완료하였다. 그 후, 15:00시경에 200고지를 점령하였고, 다양한 무기와 장비를 노획하였다. 9월 16일과 17일에는 독립 제1유격대대가 여러 위치에서 북한군과의 전투를 벌였으며, 미 구축함과의 연락 두절로 인해 추가적인
... 어려움을 겪었다.
...
... 장사동에서 위급한 상황에 처한 독립 제1유격대대를 구출하기 위해 해군본부는 LT-1(인왕호)를 급파했으나, LST 문산호의 구출에 실패했다. 해군본부는 상륙부대의 철수를 지원하기 위해 LST 조치원호를 현지로 보냈다. 9월 18일,
... 이명흠 부대장은 유엔 해군과의 협력 하에 부족한 식량과 탄약 지원을 받았다. 9월 19일, 유엔군의 함포지원과 함께 LST 조치원호가 도착하여 철수 작전을 시작했다. 스피어 소령은 직접 해안에 상륙하여 구조작전을 지시하였다. 9월 2
... 0일, 725명이 부산항으로 복귀했으나, 32명이 장사동 해안에 남아 북한군의 포로가 되었거나 탈출하여 국군에 합류하였다.
...
... 장사리 전투가 인천 상륙작전의 양동작전으로 알려졌으나, 이 전투가 드라마틱한 요소로 인해 과장되었으며, 실제로는 인천 상륙작전과 큰 관련이 없다. 또한, 북한이나 중국의 군사적 상황을 고려할 때, 장사리에서의 전투가 낙동강 전선에 영
... 향을 끼칠 가능성은 낮다.
...
... ---
...
... 문제
... 1. 지문에 나오는 지명을 모두 쓰시오.
... 2. 그중 대게로 유명한 곳은?
지문에 나오는 지명은 다음과 같습니다:
- 포항
- 장사동
- 영덕군
- 부산
- 문산호
- 조치원호
- 스피어 소령
- 낙동강 전선
대게로 유명한 곳은 영덕군입니다.

ollama 쉘에서 나올때는 Use Ctrl + d or /bye to exit.

2021년형 14인치 MacBook Pro의 M1 Pro CPU는 10코어이고 GPU는 16코어입니다. 

2020년형 13인치 MacBook Pro의 M1 CPU는 8코어이고 GPU는 8코어입니다. (저는 이겁니다)

cpu를 100% 까지 사용하면서 일했습니다. (수고했어)

LangServe로 모델 서빙

langserve_ollama % ll app
total 40
drwxr-xr-x@  8 dongsik  staff   256 Apr 19 16:35 .
drwxr-xr-x@ 12 dongsik  staff   384 Apr 20 09:22 ..
-rw-r--r--@  1 dongsik  staff     0 Apr 19 16:35 __init__.py
-rw-r--r--@  1 dongsik  staff   549 Apr 19 16:35 chain.py
-rw-r--r--@  1 dongsik  staff   723 Apr 19 16:35 chat.py
-rw-r--r--@  1 dongsik  staff   328 Apr 19 16:35 llm.py
-rw-r--r--@  1 dongsik  staff  1444 Apr 19 16:35 server.py
-rw-r--r--@  1 dongsik  staff   559 Apr 19 16:35 translator.py
(llm311) dongsik@dongsikleeui-MacBookPro langserve_ollama %

chat.py, chain.py, llm.py, translator.py 세개 파일의 llm 모델명을 내 환경에 맞게 수정합니다.

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
#llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

server.py 실행

(llm311) dongsik@dongsikleeui-MacBookPro langserve_ollama % cd app
(llm311) dongsik@dongsikleeui-MacBookPro app % pwd
/Users/dongsik/GitHub/teddylee777/langserve_ollama/app
(llm311) dongsik@dongsikleeui-MacBookPro app % ll
total 40
drwxr-xr-x@  8 dongsik  staff   256 Apr 19 16:35 .
drwxr-xr-x@ 12 dongsik  staff   384 Apr 20 09:22 ..
-rw-r--r--@  1 dongsik  staff     0 Apr 19 16:35 __init__.py
-rw-r--r--@  1 dongsik  staff   584 Apr 20 13:15 chain.py
-rw-r--r--@  1 dongsik  staff   758 Apr 20 13:15 chat.py
-rw-r--r--@  1 dongsik  staff   363 Apr 20 13:15 llm.py
-rw-r--r--@  1 dongsik  staff  1444 Apr 19 16:35 server.py
-rw-r--r--@  1 dongsik  staff   594 Apr 20 13:15 translator.py
(llm311) dongsik@dongsikleeui-MacBookPro app % python server.py

http://0.0.0.0:8000/prompt/playground/

질문 과 답변

RemoteRunable로 LangServe를 호출 하도록 변경

<경로>/langserve_ollama/example

% ll
total 120
drwxr-xr-x@  9 dongsik  staff    288 Apr 20 13:50 .
drwxr-xr-x@ 12 dongsik  staff    384 Apr 20 09:22 ..
drwxr-xr-x@  3 dongsik  staff     96 Apr 19 16:35 .streamlit
-rw-r--r--@  1 dongsik  staff  12504 Apr 19 16:35 00-ollama-test.ipynb
-rw-r--r--@  1 dongsik  staff   4885 Apr 19 16:35 01-remote-invoke.ipynb
-rw-r--r--@  1 dongsik  staff   3775 Apr 19 16:35 02-more-examples.ipynb
-rw-r--r--@  1 dongsik  staff   6222 Apr 19 16:35 main.py
-rw-r--r--@  1 dongsik  staff  14708 Apr 19 16:35 requirements.txt

01-remote-invoke.ipynb의 로컬 LangServe 주소로 변경합니다

from langserve import RemoteRunnable

# ngrok remote 주소 설정

#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
# chain = RemoteRunnable("https://poodle-deep-marmot.ngrok-free.app/prompt/")
chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")

for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
    print(token, end="")

ngrok을 이용해서 로컬 LangServe 를 Port Forwarding하기 

ngrok 가입 

https://dashboard.ngrok.com/cloud-edge/domains

M1용 설치 파일을 다운로드 받아서 설치합니다.

https://dashboard.ngrok.com/get-started/setup/macos

무료 도메인 설정 

humble-curiously-antelope.ngrok-free.app

LangServe 구동된 포트로 ngok 도메인 지정해서 포트 포워딩

ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

% ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

ngrok                                                                                                                    (Ctrl+C to quit)

K8s Gateway API support available now: https://ngrok.com/r/k8sgb

Session Status                online
Account                       dongsik.lee (Plan: Free)
Version                       3.8.0
Region                        Japan (jp)
Latency                       45ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    https://humble-curiously-antelope.ngrok-free.app -> http://localhost:8000

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

https://humble-curiously-antelope.ngrok-free.app/prompt/playground/

ngrok url로 질의를 해보면 local 서버의 GPU가 100%로 올라가면서 Output을 만들고있습니다.

01-remote-invoke.ipynb 파일의 RemoteRunnable 주소를 ngrok 주소로 변경하고 vscode로 실행해봅니다.

from langserve import RemoteRunnable

# ngrok remote 주소 설정

#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
chain = RemoteRunnable("https://humble-curiously-antelope.ngrok-free.app/prompt/")
#chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")

for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
    print(token, end="")

잘 작동됩니다.

추가 예제 

번역기 

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

# 프롬프트 설정
prompt = ChatPromptTemplate.from_template(
    "Translate following sentences into Korean:\n{input}"
)

# LangChain 표현식 언어 체인 구문을 사용합니다.
chain = prompt | llm | StrOutputParser()

LLM을 Runable로 실행

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

Streamlit으로 PDF rag 해보기

Embedding을 OpenAIEmbeddings을 사용하기위해서 OPENAI_API_KEY를 .env 파일에서 가져옵니다.

% pip install python-dotenv

main.py 내용중 OPEN API KEY세팅과 LANGSERVE_ENDPOINT를 ngrok주소로 업데이트 한후 실행합니다

% streamlit run main.py

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://192.168.0.10:8501

  For better performance, install the Watchdog module:

  $ xcode-select --install
  $ pip install watchdog

예제 > SPRI_AI_Brief_2023년12월호_F.pdf

https://spri.kr/posts?code=AI-Brief

FileNotFoundError: [Errno 2] No such file or directory: 'pdfinfo'

% conda install poppler
Channels:
 - defaults
 - conda-forge
Platform: osx-arm64

% pip install pdftotext

FileNotFoundError: [Errno 2] No such file or directory: 'tesseract'

% brew install tesseract
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

% brew install tesseract-lang

UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-23: ordinal not in range(128)

위 PDF에서 최종 질문 을 해보겠습니다.

실제해보니 내용이 엄청난 영상입니다. 

- Ollama

- EEVE 양자화 모델

- LangServe

- ngrok

- Streamlit RAG

- Asitop

감사합니다.

Asitop으로 내 M1 상태 모니터링

% pip install asitop

% sudo asitop 

sudo 패스워드 입력

앞에서 Ollama로 Gemma 경량화 모델을 실행해봤습니다. 이번엔 한글을 지원하는 경량화 모델중에 

https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF

사이트에 설치 방법과 테스트 방법이 잘 기술되어있습니다. 다만, GPU Cuda 버전이 안맞을경우 가이드대로 실행할때 Execption이 발생해서

• Original model is yanolja/EEVE-Korean-Instruct-10.8B-v1.0
• quantized using llama.cpp

llama.cpp는 4비트 정수 양자화를 이용해서 Llama 모델과 Python이 함께 실행하는(저수준 액세스 바인더) 것을 목표로 만들어진 프로젝트입니다. 의존성 없는 순수 C/C++를 통해서 구현되었으며, Mac OS, Windows, Linux 모두 실행 가능합니다.

다운로드 가능한 세가지 모델의 비교입니다

셋중에서 가장작은 Q4 모델을 다운로드 받습니다.

https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/resolve/main/ggml-model-Q4_K_M.gguf?download=true

그리고 modelfile 파일을 만들어서 다운로드 받은 모델을 Ollama에 등록해주면 됩니다.

다운로드 받는 GGML은 Apple M1 및 M2 실리콘에 최적화된 양자화 구현이라고 합니다.

다운로드 받은 모델파일을 Ollama에 등록해주기 위해서 Modelfile을 생성합니다.

ModelFile

FROM /Users/dongsik/GitHub/llm/eeve/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/ggml-model-Q4_K_M.gguf
​
TEMPLATE """### User:
{{ .Prompt }}
​
### Assistant:
"""
​
PARAMETER temperature 0.1
​
PARAMETER num_ctx 4096
PARAMETER stop "</s>"
PARAMETER stop "### System:"
PARAMETER stop "### User:"
PARAMETER stop "### Assistant:"

Modelfile과 다운로드받은 GGUF 파일이 준비되었습니다.

% ll
total 40
drwxr-xr-x  8 dongsik  staff   256 Apr 12 14:28 .
drwxr-xr-x  7 dongsik  staff   224 Apr 12 11:33 ..
drwxr-xr-x  5 dongsik  staff   160 Apr 11 00:11 .ipynb_checkpoints
drwxr-xr-x  3 dongsik  staff    96 Apr 10 23:55 EEVE-Korean-Instruct-10.8B-v1.0-GGUF
-rw-r--r--  1 dongsik  staff   325 Apr 12 14:27 Modelfile
-rw-r--r--  1 dongsik  staff  5957 Apr 11 00:26 ollama_eeve_gguf.ipynb

eeve 모델을 Ollama에 등록해줍니다

% ollama create eeve:q4 -f Modelfile
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
transferring model data
creating model layer
creating template layer
creating parameters layer
creating config layer
using already created layer sha256:5a79b80eb5e2eec5cf5d514dfa32187872dde1dae6a2b9c8
using already created layer sha256:c3de887d2d041bfea1bfed395834ea828839af278003269e
using already created layer sha256:e6b785eab1777ecfc57eab9a85f9b623931e6f1079ae6d75
using already created layer sha256:8b03799cdb5862e5cdfda70f0e116193aa07f2309015a158
writing manifest
success

성공적으로 등록되면 모델을 확인가능합니다.

gemma:2b 모델과 eeve:q4 모델 두개가 등록된것을 확인합니다.

% ollama list
NAME       	ID          	SIZE  	MODIFIED
eeve:q4    	68f4c2c2d9fe	6.5 GB	8 seconds ago
gemma:2b   	b50d6c999e59	1.7 GB	2 days ago

지울때는 rm 명령을 사용합니다 (ollama rm eeve:q4)

cli로 ollama eeve:q4 모델에 질문을 해봅니다. 

M1 노트북임을 감안했을때 상당히 빠르게 답변을 해주고 있고 퀄리티도 상당합니다.

아래 추가로 ChatGPT 3.5와 뤼튼의 GPT 4를 통해서 동일한 질문을 해본 결과 입니다.

eeve:q4 의 답변

% ollama run eeve:q4 하늘은 왜 파란색일까요?
하늘이 파란색인 이유는 대기 중에 있는 공기 분자와 먼지 입자들이 햇빛을 산란시키기 때문입니다. 태양빛이 지구 대기를 통과할 때, 이 작은 입자들과 상호 작용하여 빛을 다양한 방향으로 흩어지게 합니다. 우리가 하늘에서 보는 빛은 주로
파란색과 보라색의 가시광선 스펙트럼에 집중되어 있습니다.

이 현상은 빛이 공기 분자와 먼지 입자에 부딪힐 때 일어나는 산란 효과 때문입니다. 이 작은 입자들은 태양빛을 다양한 방향으로 흩어지게 하여, 우리가 하늘에서 보는 색깔을 만들어냅니다. 파란색과 보라색 빛은 다른 색들보다 더
산란되기 때문에, 우리 눈에 가장 많이 도달하여 하늘을 파랗게 보이게 합니다.

또한 대기 중에 있는 물방울이나 얼음 결정도 햇빛의 산란에 역할을 할 수 있습니다. 이 입자들이 태양빛과 상호 작용할 때, 빛을 다양한 방향으로 흩어지게 하여 하늘을 더욱 파랗게 보이게 만듭니다.

하늘이 항상 파란색인 것은 아니며, 대기 조건에 따라 색깔이 변할 수 있다는 점을 주목하는 것이 중요합니다. 예를 들어, 일출이나 일몰 때는 대기 중에 있는 먼지 입자와 물방울의 양이 증가하여 햇빛을 더 많이 산란시켜 하늘에 주황색,
분홍색 또는 보라색을 만들 수 있습니다.

요약하자면, 하늘이 파란색인 이유는 공기 분자, 먼지 입자 및 대기 중 물방울이나 얼음 결정과 같은 다양한 요소가 태양빛의 산란에 기여하기 때문입니다. 이 상호 작용은 파란색과 보라색 빛의 높은 농도를 만들어내어 우리가 하늘에서 보는
색깔을 만들어냅니다.

GPT-3.5의 답변

뤼튼 GPT-4의 답변

Ollama Langchain을 이용해서 추론을 잘하는지 질문을 해보겠습니다.

import time
import langchain
from langchain_community.llms import Ollama
import logging

# Configure basic logging
logging.basicConfig(level=logging.INFO)

try:
    llm = Ollama(model="eeve:q4")
    
    # 프롬프트가 잘 정의되어 있는지 확인하는 것이 필요합니다. (모델의 기능에 따라 조정가능)
    start = time.time()
    prompt = ("한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주")
    
    response = llm.invoke(prompt)
    print(response)
    print(time.time() - start)
    
except ImportError:
    logging.error("Failed to import Ollama from langchain_community. Is the package installed?")
except Exception as e:
    logging.error(f"An unexpected error occurred: {e}")

정답은 (D) 서울입니다.
3.465108871459961