!curl -fsSL https://ollama.com/install.sh | sh
>>> Installing ollama to /usr/local
>>> Downloading Linux amd64 bundle
############################################################################################# 100.0%
>>> Creating ollama user...
>>> Adding ollama user to video group...
>>> Adding current user to ollama group...
>>> Creating ollama systemd service...
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.
Ollama 시작 (실행)
nohup을 사용하여 백그라운드에서 Ollama를 시작합니다. API 서비스 포트는 11434 입니다.
!nohup ollama serve > ollama.log &
nohup: redirecting stderr to stdout
Ollama 모델 목록 확인
!ollama list
NAME ID SIZE MODIFIED
추론
명령행(command line)을 이용해서 gemma 7B 모델을 받고 추론을 실행
!ollama run gemma:7b "대한민국의 수도는 어디야?" 2> ollama.log
대한민국의 수도는 서울입니다.</end_of_turn>
영어로 질문해봅니다.
ollama run gemma:7b "What is the capital of Korea?" 2> ollama.log
The capital of Korea is **Seoul**. It is the largest and most populous city in the country.
실행한 모델 확인
!ollama list
NAME ID SIZE MODIFIED
gemma:7b a72c7f4d0a15 5.0 GB 41 seconds ago
M1 교체 이전에 사용하던 MacBook Pro 2015에서 Ollama로 Gemma2를 구동해봤습니다.
MacBook Pro 2015
Processor:Multi-core CPU (Intel i5/i7/i9 or AMD equivalent) Memory:You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models. Storage:At least 20 GB of free space on an SSD Operating System:Recent versions of Windows, macOS, or Linux
Python 3.11.4로 생성된 Conda 가상환경으로 실행합니다.
(base) Gabrielui-MacBookPro:~ gabriel$ conda env list
# conda environments:
#
base * /usr/local/anaconda3
gabriel /usr/local/anaconda3/envs/gabriel
(base) Gabrielui-MacBookPro:~ gabriel$ conda activate gabriel
(gabriel) Gabrielui-MacBookPro:~ gabriel$
(gabriel) Gabrielui-MacBookPro:~ gabriel$ python -V
Python 3.11.4
(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama --version
ollama version is 0.3.9
(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama run gemma2
pulling manifest
pulling ff1d1fc78170... 100% ▕█████████████████████████████████████████████████████████████████▏ 5.4 GB
pulling 109037bec39c... 100% ▕█████████████████████████████████████████████████████████████████▏ 136 B
pulling 097a36493f71... 100% ▕█████████████████████████████████████████████████████████████████▏ 8.4 KB
pulling 2490e7468436... 100% ▕█████████████████████████████████████████████████████████████████▏ 65 B
pulling 10aa81da732e... 100% ▕█████████████████████████████████████████████████████████████████▏ 487 B
verifying sha256 digest
writing manifest
success
>>>
다운로드한 모델을 확인합니다.
(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME ID SIZE MODIFIED
gemma2:latest ff02c3702f32 5.4 GB 14 minutes ago
(gabriel) Gabrielui-MacBookPro:~ gabriel$
Ollama + Gemma2 (9B)
ollama run gemma2 로 다운로드한 모델의 정보를 확인합니다.
>>> /show info
Model
arch gemma2
parameters 9.2B
quantization Q4_0
context length 8192
embedding length 3584
Parameters
stop "<start_of_turn>"
stop "<end_of_turn>"
License
Gemma Terms of Use
Last modified: February 21, 2024
>>>
프롬프트로 질문 "why is the sky blue?"
>>> why is the sky blue?
The sky appears blue due to a phenomenon called **Rayleigh scattering**.
Here's a breakdown:
* **Sunlight:** Sunlight is made up of all the colors of the rainbow.
* **Atmosphere:** When sunlight enters Earth's atmosphere, it collides with tiny air molecules (mainly nitrogen and
oxygen).
* **Scattering:** These molecules scatter the sunlight in all directions.
* **Wavelengths:** Blue light has a shorter wavelength than other colors in the visible spectrum. Shorter wavelengths are
scattered more effectively by the air molecules.
Therefore, blue light gets scattered much more than other colors, making the sky appear blue to our eyes.
**At sunrise and sunset:** The sunlight travels through more atmosphere to reach us. This means even more blue light is
scattered away, allowing longer wavelengths like orange and red to dominate, resulting in those beautiful hues.
>>> Send a message (/? for help)
Use Ctrl + d or /bye to exit.
(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME ID SIZE MODIFIED
gemma:2b b50d6c999e59 1.7 GB 3 minutes ago
gemma2:latest ff02c3702f32 5.4 GB 40 minutes ago
(gabriel) Gabrielui-MacBookPro:~ gabriel$
Gemma2와 동일한 질문을 합니다. "why is the sky blue?"
>>> why is the sky blue?
The sky appears blue due to Rayleigh scattering.
**Rayleigh scattering** is a scattering phenomenon where light waves are scattered in different directions depending on their wavelength.
Blue light has a shorter wavelength than other colors of light, meaning it scatters more effectively. This is why the sky appears blue.
Here's a more detailed explanation:
* **Sunlight** contains all colors of the rainbow.
* **Blue light** has a shorter wavelength than other colors.
* **When blue light** interacts with molecules in the atmosphere, it **scatters more strongly** than other colors.
* This scattering causes blue light to be scattered **towards the observer**.
* The scattered blue light is what we see in the sky as **blue color**.
**Other factors that contribute to the blue color of the sky:**
* **Dust particles:** Air molecules can also scatter light, but their scattering effect is generally weaker than that of molecules in the
atmosphere.
* **Water droplets:** Water droplets in the atmosphere can also scatter light, but their effect is typically less significant than that of
molecules.
* **Sun's position:** The sky appears blue because the sun is lower in the sky, meaning it is directly overhead. This means that more blue
light reaches our eyes.
In summary, the blue color of the sky is due to Rayleigh scattering of light by molecules in the atmosphere, which primarily results from
the shorter wavelength of blue light.
>>> Send a message (/? for help)
실행시에 MacBook의 성능을 확인합니다.
실행한 모델 정보 확인
>>> /show info
Model
arch gemma
parameters 3B
quantization Q4_0
context length 8192
embedding length 2048
Parameters
repeat_penalty 1
stop "<start_of_turn>"
stop "<end_of_turn>"
License
Gemma Terms of Use
Last modified: February 21, 2024
>>> Send a message (/? for help)
(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME ID SIZE MODIFIED
gemma:7b a72c7f4d0a15 5.0 GB 2 weeks ago
gemma:2b b50d6c999e59 1.7 GB 2 weeks ago
gemma2:latest ff02c3702f32 5.4 GB 2 weeks ago
Gemma2와 동일한 질문을 합니다. "why is the sky blue?"
>>> why is the sky blue?
**The sky is blue due to a phenomenon called Rayleigh scattering.**
* Sunlight is composed of all the colors of the rainbow, each with a different wavelength.
* Shorter wavelengths of light, such as blue light, have higher energy and more tightly spaced photons.
* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the shorter wavelengths are
scattered more effectively.
**Here's how it works:**
1. Sunlight enters the atmosphere and interacts with molecules of nitrogen and oxygen.
2. These molecules have electrons that can be excited by the energy of the incoming light.
3. The electrons absorb the energy and vibrate, then emit it in all directions.
4. However, **blue light is scattered more efficiently than other colors** because its shorter wavelength allows for more
frequent collisions between the photons and the molecules.
**This scattering process results in:**
* More blue light is scattered in all directions, reaching our eyes and making the sky appear blue.
* Longer wavelengths of light, such as red light, are scattered less efficiently, so we see more of them directly from
the sun.
**Additional factors influencing the sky's color:**
* **Time of day:** The sky is generally bluer at midday when the sunlight travels through less atmosphere.
* **Cloud cover:** Clouds can block the sunlight and scatter less blue light, making the sky appear less blue.
* **Pollution:** Pollution in the atmosphere can also scatter light and affect the sky's color.
>>> Send a message (/? for help)
실행한 모델 정보 확인
>>> /show info
Model
arch gemma
parameters 9B
quantization Q4_0
context length 8192
embedding length 3072
Parameters
penalize_newline false
repeat_penalty 1
stop "<start_of_turn>"
stop "<end_of_turn>"
License
Gemma Terms of Use
Last modified: February 21, 2024
>>> Send a message (/? for help)
질의 성능 확인
모델 실행 성능 확인
Intel CPU의 MacBook에서 Gemma2 9B 모델 그리고 Gemma 2B, 7B 모델까지 Ollama로 구동해 봤습니다.
Huggingface에서 모델 Download 받고 Ollama에 EEVE Q5 모델 등록하고 구동
huggingface-cli download \
heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF \
ggml-model-Q5_K_M.gguf \
--local-dir /Users/dongsik/GitHub/teddylee777/langserve_ollama/ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0 \
--local-dir-use-symlinks False
Consider using `hf_transfer` for faster downloads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.
downloading https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/resolve/main/ggml-model-Q5_K_M.gguf to /Users/dongsik/.cache/huggingface/hub/tmpkuuur4ki
ggml-model-Q5_K_M.gguf: 37%|███████████████████████████▌ | 2.81G/7.65G [04:35<09:55, 8.13MB/s]
FROM ggml-model-Q5_K_M.gguf
TEMPLATE """{{- if .System }}
<s>{{ .System }}</s>
{{- end }}
<s>Human:
{{ .Prompt }}</s>
<s>Assistant:
"""
SYSTEM """A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."""
PARAMETER TEMPERATURE 0
PARAMETER stop <s>
PARAMETER stop </s>
모델파일 설정하지 않으면 답변이 끝났을때 이상하게 대답할수도있기때문에 필요합니다.
System prompt가 있다면 중간(.System) 위치에 넣어으라는 의미이며 여기서는 'SYSTEM'이 이자리를 치환하게 됩니다.
그다음 <s> 스페셜 토큰이 앞에 붙어서 사용자 즉 Human의 질문 .Prompt가 들어가게 됩니다.
그후 모델 Assistant가 받아서 답변하게 됩니다.
※ Note!!
Modelfile에서 <s>는 문장의 시작을 나타내는 특수 토큰입니다. 이것은 "문장의 시작"을 나타내기 위해 사용됩니다. 예를 들어, 자연어 처리 작업에서 모델이 문장의 시작을 식별하고, 이에 따라 적절한 처리를 수행할 수 있도록 합니다. 이것은 토큰화된 데이터의 일부로서 모델에 제공됩니다.
tokenizer.chat_template
{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful assistant.' %}{% endif %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{{'<|im_start|>system ' + system_message + '<|im_end|> '}}{% endif %}{{'<|im_start|>' + message['role'] + ' ' + message['content'] + '<|im_end|>' + ' '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant ' }}{% endif %}
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: {prompt}
Assistant:
ollama 목록 확인
% ollama list
NAME ID SIZE MODIFIED
eeve:q4 68f4c2c2d9fe 6.5 GB 7 days ago
gemma:2b b50d6c999e59 1.7 GB 10 days ago
저는 위의 Modelfile로 ollama등록할려고 하니 "Error: unknown parameter 'TEMPERATURE'"가 발생했습니다. 그래서 소문자 temperature로 변경해서 생성되었습니다. 만일 동일한 에러가 발생한다면 소문자 temperature로 변경해서 생성해보시기 바랍니다.
% ollama create eeve:q5 -f ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile
transferring model data
creating model layer
creating template layer
creating system layer
creating parameters layer
creating config layer
using already created layer sha256:b9e3d1ad5e8aa6db09610d4051820f06a5257b7d7f0b06c00630e376abcfa4c1
writing layer sha256:6b70a2ad0d545ca50d11b293ba6f6355eff16363425c8b163289014cf19311fc
writing layer sha256:1fa69e2371b762d1882b0bd98d284f312a36c27add732016e12e52586f98a9f5
writing layer sha256:3ab8c1bbd3cd85e1b39b09f5ff9a76e64da20ef81c22ec0937cc2e7076f1a81c
writing layer sha256:d86595b443c06710a3e5ba27700c6a93ded80100ff1aa808a7f3444ff529fa70
writing manifest
success
% ollama list
NAME ID SIZE MODIFIED
eeve:q4 68f4c2c2d9fe 6.5 GB 7 days ago
eeve:q5 0732d4a47219 7.7 GB 7 minutes ago
gemma:2b b50d6c999e59 1.7 GB 10 days ago
ollama run eeve:q5
% ollama run eeve:q5
>>> 대한민국의 수도는 어디야?
안녕하세요! 대한민국의 수도에 대해 궁금해하시는군요. 서울이 바로 그 곳입니다! 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를 가진 대도시로,
세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며, 많은 바와 클럽
관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?
>>>
로컬 command로 실행한 모습
아래 문구로 질문해보겠습니다.
한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주
>>> 한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주
대한민국의 수도에 대한 질문에 답변해 주셔서 감사합니다! 정답은 (D) 서울입니다. 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를
대도시로, 세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며,
많은 바와 클럽이 관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?
질문과 동시에 답변이 나오는것처럼 작동합니다. 속도도 좋고 답변의 퀄리티도 좋습니다.
>>> 다음 지문을 읽고 문제에 답하시오.
...
... ---
...
... 1950년 7월, 한국 전쟁 초기에 이승만 대통령은 맥아더 장군에게 유격대원들을 북한군의 후방에 침투시키는 방안을 제안했다. 이후, 육군본부는 육본직할 유격대와 육본 독립 유격대를 편성했다. 국군은 포항과 인접한 장사동 지역에 상륙작
... 전을 수행할 부대로 독립 제1유격대대를 선정했다. 육군본부는 독립 제1유격대대에 동해안의 장사동 해안에 상륙작전을 감행하여 북한군 제2군단의 보급로를 차단하고 국군 제1군단의 작전을 유리하게 하기 위한 작전명령(육본 작명 제174호)
... 을 하달했다. 9월 14일, 독립 제1유격대대는 부산에서 LST 문산호에 승선하여 영덕군의 장사동으로 출항했다.
...
... 1950년 9월 15일, 독립 제1유격대대는 장사동 해안에 상륙을 시도하였으나 태풍 케지아로 인한 높은 파도와 안개로 인해 어려움을 겪었다. LST 문산호는 북한군의 사격과 파도로 인해 좌초되었고, 상륙부대는 09:00시경에 전원이
... 상륙을 완료하였다. 그 후, 15:00시경에 200고지를 점령하였고, 다양한 무기와 장비를 노획하였다. 9월 16일과 17일에는 독립 제1유격대대가 여러 위치에서 북한군과의 전투를 벌였으며, 미 구축함과의 연락 두절로 인해 추가적인
... 어려움을 겪었다.
...
... 장사동에서 위급한 상황에 처한 독립 제1유격대대를 구출하기 위해 해군본부는 LT-1(인왕호)를 급파했으나, LST 문산호의 구출에 실패했다. 해군본부는 상륙부대의 철수를 지원하기 위해 LST 조치원호를 현지로 보냈다. 9월 18일,
... 이명흠 부대장은 유엔 해군과의 협력 하에 부족한 식량과 탄약 지원을 받았다. 9월 19일, 유엔군의 함포지원과 함께 LST 조치원호가 도착하여 철수 작전을 시작했다. 스피어 소령은 직접 해안에 상륙하여 구조작전을 지시하였다. 9월 2
... 0일, 725명이 부산항으로 복귀했으나, 32명이 장사동 해안에 남아 북한군의 포로가 되었거나 탈출하여 국군에 합류하였다.
...
... 장사리 전투가 인천 상륙작전의 양동작전으로 알려졌으나, 이 전투가 드라마틱한 요소로 인해 과장되었으며, 실제로는 인천 상륙작전과 큰 관련이 없다. 또한, 북한이나 중국의 군사적 상황을 고려할 때, 장사리에서의 전투가 낙동강 전선에 영
... 향을 끼칠 가능성은 낮다.
...
... ---
...
... 문제
... 1. 지문에 나오는 지명을 모두 쓰시오.
... 2. 그중 대게로 유명한 곳은?
지문에 나오는 지명은 다음과 같습니다:
- 포항
- 장사동
- 영덕군
- 부산
- 문산호
- 조치원호
- 스피어 소령
- 낙동강 전선
대게로 유명한 곳은 영덕군입니다.
ollama 쉘에서 나올때는 Use Ctrl + d or /bye to exit.
2021년형 14인치 MacBook Pro의 M1 Pro CPU는 10코어이고 GPU는 16코어입니다.
from langserve import RemoteRunnable
# ngrok remote 주소 설정
#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
# chain = RemoteRunnable("https://poodle-deep-marmot.ngrok-free.app/prompt/")
chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")
for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
print(token, end="")
01-remote-invoke.ipynb 파일의 RemoteRunnable 주소를 ngrok 주소로 변경하고 vscode로 실행해봅니다.
from langserve import RemoteRunnable
# ngrok remote 주소 설정
#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
chain = RemoteRunnable("https://humble-curiously-antelope.ngrok-free.app/prompt/")
#chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")
for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
print(token, end="")
잘 작동됩니다.
추가 예제
번역기
from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")
# 프롬프트 설정
prompt = ChatPromptTemplate.from_template(
"Translate following sentences into Korean:\n{input}"
)
# LangChain 표현식 언어 체인 구문을 사용합니다.
chain = prompt | llm | StrOutputParser()
LLM을 Runable로 실행
from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")
main.py 내용중 OPEN API KEY세팅과 LANGSERVE_ENDPOINT를 ngrok주소로 업데이트 한후 실행합니다
% streamlit run main.py
You can now view your Streamlit app in your browser.
Local URL: http://localhost:8501
Network URL: http://192.168.0.10:8501
For better performance, install the Watchdog module:
$ xcode-select --install
$ pip install watchdog
FileNotFoundError: [Errno 2] No such file or directory: 'tesseract'
% brew install tesseract
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
% brew install tesseract-lang
UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-23: ordinal not in range(128)
llama.cpp는 4비트 정수 양자화를 이용해서 Llama 모델과 Python이 함께 실행하는(저수준 액세스 바인더) 것을 목표로 만들어진 프로젝트입니다. 의존성 없는 순수 C/C++를 통해서 구현되었으며, Mac OS, Windows, Linux 모두 실행 가능합니다.
다운로드 가능한 세가지 모델의 비교입니다
GGUF
ggml-model-Q4_K_M.gguf
ggml-model-Q5_K_M.gguf
ggml-model-f16.gguf
Size
6.51 GB
7.65 GB
21.6 GB
Metadata
Value
Value
Value
version
3
3
3
tensor_count
435
435
435
kv_count
24
24
23
general.architecture
llama
llama
llama
general.name
LLaMA v2
LLaMA v2
LLaMA v2
general.file_type
15
17
1
general.quantization_version
2
2
4096
llama.context_length
4096
4096
4096
llama.embedding_length
4096
4096
48
llama.block_count
48
48
14336
llama.feed_forward_length
14336
14336
128
llama.rope.dimension_count
128
128
10000
llama.rope.freq_base
10000
10000
32
llama.attention.head_count
32
32
8
llama.attention.head_count_kv
8
8
1E-05
llama.attention.layer_norm_rms_epsilon
1E-05
1E-05
-
tokenizer.ggml.model
llama
llama
llama
tokenizer.ggml.tokens
[<unk>, <s>, </s>, <0x00>, <0x01>, ...]
[<unk>, <s>, </s>, <0x00>, <0x01>, ...]
[<unk>, <s>, </s>, <0x00>, <0x01>, ...]
tokenizer.ggml.scores
[-1000, -1000, -1000, -1000, -1000, ...]
[-1000, -1000, -1000, -1000, -1000, ...]
[-1000, -1000, -1000, -1000, -1000, ...]
tokenizer.ggml.token_type
[3, 3, 3, 6, 6, ...]
[3, 3, 3, 6, 6, ...]
[3, 3, 3, 6, 6, ...]
tokenizer.ggml.bos_token_id
1
1
1
tokenizer.ggml.eos_token_id
32000
32000
32000
tokenizer.ggml.unknown_token_id
0
0
0
tokenizer.ggml.padding_token_id
2
2
2
tokenizer.ggml.add_bos_token
TRUE
TRUE
TRUE
tokenizer.ggml.add_eos_token
FALSE
FALSE
FALSE
tokenizer.chat_template
{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful assistant.' %}{% endif %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{{'<|im_start|>system ' + system_message + '<|im_end|> '}}{% endif %}{{'<|im_start|>' + message['role'] + ' ' + message['content'] + '<|im_end|>' + ' '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant ' }}{% endif %}
{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful assistant.' %}{% endif %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{{'<|im_start|>system ' + system_message + '<|im_end|> '}}{% endif %}{{'<|im_start|>' + message['role'] + ' ' + message['content'] + '<|im_end|>' + ' '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant ' }}{% endif %}
{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{% set system_message = messages[0]['content'] %}{% else %}{% set loop_messages = messages %}{% set system_message = 'You are a helpful assistant.' %}{% endif %}{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{{'<|im_start|>system ' + system_message + '<|im_end|> '}}{% endif %}{{'<|im_start|>' + message['role'] + ' ' + message['content'] + '<|im_end|>' + ' '}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant ' }}{% endif %}
% ollama create eeve:q4 -f Modelfile
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
transferring model data
creating model layer
creating template layer
creating parameters layer
creating config layer
using already created layer sha256:5a79b80eb5e2eec5cf5d514dfa32187872dde1dae6a2b9c8
using already created layer sha256:c3de887d2d041bfea1bfed395834ea828839af278003269e
using already created layer sha256:e6b785eab1777ecfc57eab9a85f9b623931e6f1079ae6d75
using already created layer sha256:8b03799cdb5862e5cdfda70f0e116193aa07f2309015a158
writing manifest
success
성공적으로 등록되면 모델을 확인가능합니다.
gemma:2b 모델과 eeve:q4 모델 두개가 등록된것을 확인합니다.
% ollama list
NAME ID SIZE MODIFIED
eeve:q4 68f4c2c2d9fe 6.5 GB 8 seconds ago
gemma:2b b50d6c999e59 1.7 GB 2 days ago
지울때는 rm 명령을 사용합니다 (ollama rm eeve:q4)
cli로 ollama eeve:q4 모델에 질문을 해봅니다.
M1 노트북임을 감안했을때 상당히 빠르게 답변을 해주고 있고 퀄리티도 상당합니다.
아래 추가로 ChatGPT 3.5와 뤼튼의 GPT 4를 통해서 동일한 질문을 해본 결과 입니다.
eeve:q4 의 답변
% ollama run eeve:q4 하늘은 왜 파란색일까요?
하늘이 파란색인 이유는 대기 중에 있는 공기 분자와 먼지 입자들이 햇빛을 산란시키기 때문입니다. 태양빛이 지구 대기를 통과할 때, 이 작은 입자들과 상호 작용하여 빛을 다양한 방향으로 흩어지게 합니다. 우리가 하늘에서 보는 빛은 주로
파란색과 보라색의 가시광선 스펙트럼에 집중되어 있습니다.
이 현상은 빛이 공기 분자와 먼지 입자에 부딪힐 때 일어나는 산란 효과 때문입니다. 이 작은 입자들은 태양빛을 다양한 방향으로 흩어지게 하여, 우리가 하늘에서 보는 색깔을 만들어냅니다. 파란색과 보라색 빛은 다른 색들보다 더
산란되기 때문에, 우리 눈에 가장 많이 도달하여 하늘을 파랗게 보이게 합니다.
또한 대기 중에 있는 물방울이나 얼음 결정도 햇빛의 산란에 역할을 할 수 있습니다. 이 입자들이 태양빛과 상호 작용할 때, 빛을 다양한 방향으로 흩어지게 하여 하늘을 더욱 파랗게 보이게 만듭니다.
하늘이 항상 파란색인 것은 아니며, 대기 조건에 따라 색깔이 변할 수 있다는 점을 주목하는 것이 중요합니다. 예를 들어, 일출이나 일몰 때는 대기 중에 있는 먼지 입자와 물방울의 양이 증가하여 햇빛을 더 많이 산란시켜 하늘에 주황색,
분홍색 또는 보라색을 만들 수 있습니다.
요약하자면, 하늘이 파란색인 이유는 공기 분자, 먼지 입자 및 대기 중 물방울이나 얼음 결정과 같은 다양한 요소가 태양빛의 산란에 기여하기 때문입니다. 이 상호 작용은 파란색과 보라색 빛의 높은 농도를 만들어내어 우리가 하늘에서 보는
색깔을 만들어냅니다.
GPT-3.5의 답변
뤼튼 GPT-4의 답변
Ollama Langchain을 이용해서 추론을 잘하는지 질문을 해보겠습니다.
import time
import langchain
from langchain_community.llms import Ollama
import logging
# Configure basic logging
logging.basicConfig(level=logging.INFO)
try:
llm = Ollama(model="eeve:q4")
# 프롬프트가 잘 정의되어 있는지 확인하는 것이 필요합니다. (모델의 기능에 따라 조정가능)
start = time.time()
prompt = ("한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주")
response = llm.invoke(prompt)
print(response)
print(time.time() - start)
except ImportError:
logging.error("Failed to import Ollama from langchain_community. Is the package installed?")
except Exception as e:
logging.error(f"An unexpected error occurred: {e}")
% ollama run gemma:2b
pulling manifest
pulling c1864a5eb193... 100% ▕██████████████████████████████████████████████▏ 1.7 GB
pulling 097a36493f71... 100% ▕██████████████████████████████████████████████▏ 8.4 KB
pulling 109037bec39c... 100% ▕██████████████████████████████████████████████▏ 136 B
pulling 22a838ceb7fb... 100% ▕██████████████████████████████████████████████▏ 84 B
pulling 887433b89a90... 100% ▕██████████████████████████████████████████████▏ 483 B
verifying sha256 digest
writing manifest
removing any unused layers
success
>>> hi
Hi! 👋 How can I assist you today? 😊
Is there anything I can help you with?
>>> Send a message (/? for help)
간단하게 "hi"로 인사해 봤습니다.
이제 실행창에서 나오도록 합니다 "/bye"를 입력합니다.
>>> /bye
Hello! 👋 It's nice to hear from you. How can I help you today? 😊
Is there anything I can do for you?
>>> /bye
(base) dongsik@dongsikleeui-MacBookPro ~ %
바로 내보내주지는 않으면 한번더 "/bye" 합니다
설치되어있는 모델을 확인할수도 있습니다.
% ollama list
NAME ID SIZE MODIFIED
gemma:2b b50d6c999e59 1.7 GB 6 minutes ago
또한 설치된 모델을 삭제도 할수있습니다. 설치와 삭제가 너무간단합니다.
% ollama rm gemma:2b
deleted 'gemma:2b'
% ollama list
NAME ID SIZE MODIFIED
%
그럼 다시 설치하고 간단히 사용방법을 설명합니다.
1. LangChain 으로 실행하기
# LangChain 설치
pip install langchain
import langchain
# LangChain 버전 확인
print('LangChain version:', langchain.__version__)
결과
LangChain version: 0.1.12
로컬에 설치든 ollama gemma:2b 모델을 사용하도록 설정하고 실행합니다.
from langchain_community.llms import Ollama
import logging
# logging 설정
logging.basicConfig(level=logging.INFO)
try:
llm = Ollama(model="gemma:2b")
# 프롬프트가 잘 정의되어 있는지 확인하는 것이 필요합니다. (모델의 기능에 따라 조정가능)
prompt = ("Why is the sky blue?")
response = llm.invoke(prompt)
print(response)
except ImportError:
logging.error("Failed to import Ollama from langchain_community. Is the package installed?")
except Exception as e:
logging.error(f"An unexpected error occurred: {e}")
결과 :
The sky appears blue due to Rayleigh scattering. This scattering process occurs when light interacts with molecules in the Earth's atmosphere.
* **Blue light has a longer wavelength than other colors**. This means it can penetrate further into the atmosphere.
* **Blue light waves have more energy** than other colors, so they are more likely to scatter.
* **Water vapor molecules** in the atmosphere absorb blue light more efficiently than other colors.
* **Scattered blue light** is scattered in all directions equally, giving the sky its blue color.
The amount and intensity of blue scattering depends on several factors, including:
* **Particle size and density of the particles**: Smaller particles scatter light more efficiently than larger particles.
* **The wavelength of light**: Blue light is scattered more strongly than other colors.
* **Atmospheric conditions**: Temperature, humidity, and air density can also affect scattering.
Overall, the scattering of sunlight in the atmosphere creates the blue color of the sky.
url call로 호출하고 결과를 streaming 방식으로 stand out으로 출력합니다.
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms.ollama import Ollama
llm = Ollama(
base_url="http://localhost:11434",
model="gemma:2b",
callback_manager=CallbackManager(
[StreamingStdOutCallbackHandler()],
),
)
prompt = ("Why is the sky blue?")
response = llm.invoke(prompt)
print(response)
결과 :
The sky appears blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength, such as blue light. This scattering causes longer wavelengths, such as red and yellow light, to be scattered more than blue light. As a result, the sky appears blue to us.The sky appears blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength, such as blue light. This scattering causes longer wavelengths, such as red and yellow light, to be scattered more than blue light. As a result, the sky appears blue to us.
Request
% curl http://localhost:11434/api/generate -d '{
"model": "gemma:2b",
"prompt":"Why is the sky blue?",
"stream": false
}'
Reponse
{"model":"gemma:2b","created_at":"2024-04-10T01:49:38.296228Z","response":"The sky is blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength. This means that blue light has a greater wavelength and is scattered more than other colors. This is why the sky appears blue.","done":true,"context":[106,1645,108,4385,603,573,8203,3868,235336,107,108,106,2516,108,651,8203,603,3868,3402,577,153902,38497,235265,153902,38497,603,573,38497,576,2611,731,16071,576,476,25270,35571,235265,1417,3454,674,3868,2611,919,476,6561,35571,578,603,30390,978,1178,1156,9276,235265,1417,603,3165,573,8203,8149,3868,235265,107,108],"total_duration":1936533375,"load_duration":3180292,"prompt_eval_duration":272404000,"eval_count":49,"eval_duration":1658092000}
Request
% curl http://localhost:11434/api/chat -d '{
"model": "gemma:2b",
"messages": [
{
"role": "user",
"content": "why is the sky blue?"
}
],
"stream": false
}'
Response
{"model":"gemma:2b","created_at":"2024-04-10T01:30:16.821415Z","message":{"role":"assistant","content":"The sky appears blue due to Rayleigh scattering. This phenomenon occurs when sunlight interacts with molecules in the Earth's atmosphere.\n\n**Rayleigh Scattering:**\n\n* Sunlight is composed of all colors of the spectrum, including blue, violet, yellow, orange, and red.\n* When sunlight enters the atmosphere, it interacts with molecules such as nitrogen and oxygen molecules.\n* These molecules have different sizes and structures, which cause different wavelengths of light to scatter in different directions.\n* Blue light, with its shorter wavelengths, is scattered more strongly than other colors due to its shorter path length through the atmosphere.\n\n**Blue Sky:**\n\n* As a result, blue light is scattered in all directions from the Sun.\n* This scattering effect spreads out the Sun's light throughout the atmosphere, making the sky appear blue.\n* The intensity of blue light can vary slightly depending on factors such as altitude, temperature, and atmospheric conditions.\n\n**Other Factors:**\n\n* The scattering process depends on the size and density of the molecules, which is why the sky appears blue even though the Sun is a star of much greater temperature.\n* The atmosphere is composed of different gases with varying densities, which influences the scattering process.\n* Cloud and pollution can also affect the sky's color, with clouds reflecting blue light more efficiently than other colors.\n\n**Conclusion:**\n\nThe blue color of the sky is primarily caused by Rayleigh scattering of sunlight by molecules in the Earth's atmosphere. This scattering process spreads out the Sun's light throughout the sky, making it appear blue to us on Earth."},"done":true,"total_duration":11417865583,"load_duration":4883667,"prompt_eval_duration":270041000,"eval_count":324,"eval_duration":11140497000}
