끄적끄적

이번 수요일(2025년 3월 12일)에 발표되었습니다. 1B, 4B, 12B, 27B의 4가지 크기로 출시되었으며, 사전 학습된 버전과 지침 미세 조정된 버전이 있습니다. 모델 4B, 12B, 27B는 확장된 컨텍스트 창(최대 128K)과 다중 모달리티를 제공합니다.

Macbook M1 + Ollama + gemma3 로 Multimodal 이미지 처리를 테스트해보겠습니다

Ollama Gemma3

Gemma is a lightweight, family of models from Google built on Gemini technology. The Gemma 3 models are multimodal—processing text and images—and feature a 128K context window with support for over 140 languages. Available in 1B, 4B, 12B, and 27B parameter sizes, they excel in tasks like question answering, summarization, and reasoning, while their compact design allows deployment on resource-limited devices.

Gemma는 Google이 Gemini 기술을 기반으로 개발한 경량 모델 계열입니다. Gemma 3 모델은 멀티모달 모델로, 텍스트와 이미지를 처리할 수 있으며 128K 컨텍스트 윈도우를 제공하고 140개 이상의 언어를 지원합니다. 1B, 4B, 12B, 27B 파라미터 크기로 제공되며, 질문 응답, 요약, 추론과 같은 작업에서 뛰어난 성능을 발휘합니다. 또한, 컴팩트한 설계로 인해 자원이 제한된 장치에서도 실행할 수 있습니다.

더욱 상세한 모델 설명을 보고자 한다면 "Gemma 3 기술 보고서 ​​PDF"를 참고하세요.

Models

Text

- 1B parameter model (32k context window)

Multimodal (Vision)

- 4B parameter model (128k context window)

- 12B parameter model (128k context window)

- 27B parameter model (128k context window)

Default 모델은 4B 모델이고 1B는 text 이고 4B 사이즈부터 Multimodal 모델입니다. 4B 모델을 받습니다.

ollama run gemma3:4b

dongsik@dongsikleeui-MacBookPro % ollama run gemma3:4b
pulling manifest
pulling 377655e65351... 100% ▕█████████████████████████████████████████████████▏ 3.3 GB
pulling e0a42594d802... 100% ▕█████████████████████████████████████████████████▏  358 B
pulling dd084c7d92a3... 100% ▕█████████████████████████████████████████████████▏ 8.4 KB
pulling 0a74a8735bf3... 100% ▕█████████████████████████████████████████████████▏   55 B
pulling ffae984acbea... 100% ▕█████████████████████████████████████████████████▏  489 B
verifying sha256 digest
writing manifest
success
>>> Send a message (/? for help)

설치 확인

dongsik@dongsikleeui-MacBookPro % ollama --version
ollama version is 0.6.0
dongsik@dongsikleeui-MacBookPro % ollama list
NAME                        ID              SIZE      MODIFIED
gemma3:4b                   c0494fe00251    3.3 GB    41 minutes ago
dongsik@dongsikleeui-MacBookPro %

첫번째 이미지 분석입니다. 

멋지고 쿨한 선글라스를 쓴 3마리 고양이입니다.

사진에서 어떤 설명을 하는지 먼저 볼까요? 우선 영어로 질문해보겠습니다 "Describe what you see in this picture"

dongsik@dongsikleeui-MacBookPro % ollama run gemma3:4b
>>> Describe what you see in this picture vecteezy_three-cool-cats.jpg
Okay, I've analyzed the image vecteezy_three-cool-cats.jpg. Here's a description of what I see:

**Overall Impression:**

The image is a playful and vibrant digital illustration featuring three stylized cats.
It has a cartoonish, slightly retro feel, likely intended for a fun or whimsical design.

**Detailed Description:**

*   **Subject:** There are three cats, each with a distinct style.
    *   **Cat 1 (Left):** This cat is a bright, almost neon blue with a sleek, modern design.
        It has large, expressive eyes and a confident pose, looking directly at the viewer.   
    *   **Cat 2 (Center):** This cat is a vibrant pink with a more rounded, friendly appearance.
        It has a slightly mischievous expression and is playfully leaning forward.
    *   **Cat 3 (Right):** This cat is a bright yellow with a more geometric, almost robotic-looking 
        design. It has a slightly tilted head and a curious expression.

*   **Style:** The illustration is clearly digital and stylized. 
    The lines are clean and bold, and the colors are saturated. 
    It resembles a vector graphic, which means it can be scaled up or down without losing quality.

*   **Background:** The background is a solid, dark gray, which helps the bright colors of the cats 
    to pop.

*   **Composition:** The cats are arranged in a horizontal line, with the central cat slightly closer
    to the viewer.

*   **Overall Tone:** The image has a cheerful and energetic tone. It's designed to be appealing and fun.

**In essence, it's a cute and colorful digital illustration of three stylized cats, 
  likely intended for use in a graphic design project.**

Would you like me to focus on a specific aspect of the image, such as the style, color palette, 
or potential uses for this illustration?

>>> Send a message (/? for help)

Macbook M1에서 처리속도는 꽤 빠릅니다. 최초 질문에 응답하는데 20초 정도에 위 설명을 마쳤습니다.

>>> 이 그림에서 보이는 것을 한글로 설명하세요 vecteezy_three-cool-cats.jpg
네, vecteezy_three-cool-cats.jpg 그림에 보이는 것을 설명해 드리겠습니다.

이 그림은 세 마리의 귀여운 고양이 그림입니다. 각 고양이는 다른 색깔과 스타일을 가지고 있습니다.

*   **왼쪽 고양이:** 밝은 파란색으로, 세련되고 현대적인 느낌을 줍니다. 눈이 크고 자신감 있는 표정을 짓고 있습니다.
*   **중간 고양이:** 밝은 분홍색으로, 둥글고 친근한 느낌을 줍니다. 장난기 넘치는 표정으로 앞으로 기대어 있습니다.
*   **오른쪽 고양이:** 밝은 노란색으로, 기하학적인 디자인을 하고 있습니다. 머리를 살짝 기울여 호기심 있는 표정을 짓고 있습니다.

전체적으로 디지털 그림이며, 선이 깔끔하고 색상이 강렬하게 표현되어 있습니다. 배경은 어두운 회색으로, 고양이들의 색깔을 더욱 돋보이게 합니다.

전반적으로 귀엽고 활기찬 느낌을 주는 그림입니다.

혹시 그림의 특정 부분에 대해 더 자세한 설명을 원하시면 말씀해주세요.

>>> Send a message (/? for help)

오른쪽 고양이의 썬글라스 색을 물어봤는데 제눈에는 노란색으로 보입니다. gemma3는 밝은 금색(Gold) 라고 합니다.

아마 저색이 금색이겠죠?

>>> 이 그림에서 오른쪽 고양이의 썬글라스 색깔을 알려주세요 vecteezy_three-cool-cats.jpg
오른쪽 고양이의 썬글라스 색깔은 **금색 (Gold)** 입니다.

사진을 자세히 살펴보니, 썬글라스 렌즈와 프레임이 모두 밝은 금색으로 되어 있습니다.

>>> Send a message (/? for help)

>>> Please tell me the color of the sunglasses on the cat on the right in this picture. vecteezy_three-cool-cats.jpg
The sunglasses on the cat on the right are **gold**. They have a bright, metallic gold color.

>>>

이번엔 표가 있는 이미지를 가지고 테스트 해보겠습니다.

>>> Describe the content of the given image ~/gemma3_prerequisites.jpeg
Added image '/Users/dongsik/workspace/ekyc/gemma3_prerequisites.jpeg'
The image is a table outlining the recommended GPU requirements for different model sizes (in billions of parameters) for a Text-to-Text model.
Here’s a breakdown of the information presented:

*   **Parameters (Billion):** The table lists the model size in billions of parameters (1B, 4B, 12B, 27B).
*   **VRAM Requirement (Text-to-Text):** This column specifies the amount of VRAM (Video RAM) needed to run the model for Text-to-Text tasks.
*   **VRAM Requirement (Image-to-Text):** This column specifies the amount of VRAM needed to run the model for Image-to-Text tasks.
*   **Recommended GPU:** This column lists the recommended GPU based on the VRAM requirement.

Essentially, it’s a chart showing the relationship between model size, VRAM requirements, and the corresponding GPU needed to run the model
effectively.

Do you want me to elaborate on any specific aspect of this table, such as the VRAM requirements for a particular model size or the GPU
recommended for a specific task?

>>>

>>> 주어진 이미지의 내용을 설명하세요. /Users/dongsik/workspace/ekyc/gemma3_prerequisites.jpeg
Added image '/Users/dongsik/workspace/ekyc/gemma3_prerequisites.jpeg'
다음은 이미지의 내용에 대한 설명입니다.

**표는 텍스트-투-텍스트 및 이미지-투-텍스트 작업에 필요한 GPU 사양을 나타냅니다.**

*   **Parameters (Billions):** 모델의 파라미터 수를 나타냅니다.
*   **VRAM Requirement (Text-to-Text):** 텍스트-투-텍스트 작업에 필요한 VRAM(비디오 RAM)의 양을 나타냅니다.
*   **VRAM Requirement (Image-to-Text):** 이미지-투-텍스트 작업에 필요한 VRAM의 양을 나타냅니다.
*   **Recommended GPU:** 주어진 파라미터 수에 맞는 VRAM 요구 사항을 충족하는 데 권장되는 GPU 모델을 나열합니다.

**주요 관찰 사항:**

*   파라미터 수가 증가함에 따라 텍스트-투-텍스트 및 이미지-투-텍스트 작업 모두에 필요한 VRAM 요구 사항이 크게 증가합니다.
*   더 큰 모델은 효율적으로 작동하기 위해 훨씬 더 많은 VRAM이 필요합니다.

이 표가 도움이 되었기를 바랍니다!

>>>

그럼 마지막으로 이미지에서 답을 찾아오는 질문을 해보겠습니다,

"그림에서 4B 모델을 image to text 하기위해 필요한 VRAM은 은 얼마인가요?"  네, 답은 10.4B 입니다.

결과를 보시죠.

>>> 그림에서 4B 모델을 image to text 하기위해 필요한 VRAM은 은 얼마인가요?
그림에 따르면 4B 모델을 image to text 하기 위해 필요한 VRAM은 10.4 GB입니다.

>>>

오....!!!

M1 교체 이전에 사용하던 MacBook Pro 2015에서 Ollama로 Gemma2를 구동해봤습니다.

Processor: Multi-core CPU (Intel i5/i7/i9 or AMD equivalent)
Memory: You should have at least 8 GB of RAM available to run the 7B models, 16 GB to run the 13B models, and 32 GB to run the 33B models.
Storage: At least 20 GB of free space on an SSD
Operating System: Recent versions of Windows, macOS, or Linux

Python 3.11.4로 생성된 Conda 가상환경으로 실행합니다.

(base) Gabrielui-MacBookPro:~ gabriel$ conda env list
# conda environments:
#
base                  *  /usr/local/anaconda3
gabriel                  /usr/local/anaconda3/envs/gabriel

(base) Gabrielui-MacBookPro:~ gabriel$ conda activate gabriel
(gabriel) Gabrielui-MacBookPro:~ gabriel$

(gabriel) Gabrielui-MacBookPro:~ gabriel$ python -V
Python 3.11.4

세 가지 크기: 2B, 9B 및 27B 파라미터의 모델

• 2B Parameters ollama run gemma2:2b
• 9B Parameters ollama run gemma2 (Default)
• 27B Parameters ollama run gemma2:27b

셋중에서 9B 모델이 Default 입니다. 

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama --version
ollama version is 0.3.9

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama run gemma2
pulling manifest
pulling ff1d1fc78170... 100% ▕█████████████████████████████████████████████████████████████████▏ 5.4 GB
pulling 109037bec39c... 100% ▕█████████████████████████████████████████████████████████████████▏  136 B
pulling 097a36493f71... 100% ▕█████████████████████████████████████████████████████████████████▏ 8.4 KB
pulling 2490e7468436... 100% ▕█████████████████████████████████████████████████████████████████▏   65 B
pulling 10aa81da732e... 100% ▕█████████████████████████████████████████████████████████████████▏  487 B
verifying sha256 digest
writing manifest
success
>>>

다운로드한 모델을 확인합니다.

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME         	ID          	SIZE  	MODIFIED
gemma2:latest	ff02c3702f32	5.4 GB	14 minutes ago
(gabriel) Gabrielui-MacBookPro:~ gabriel$

Ollama + Gemma2 (9B)

ollama run gemma2 로 다운로드한 모델의 정보를 확인합니다.

>>> /show info
  Model
  	arch            	gemma2
  	parameters      	9.2B
  	quantization    	Q4_0
  	context length  	8192
  	embedding length	3584

  Parameters
  	stop	"<start_of_turn>"
  	stop	"<end_of_turn>"

  License
  	Gemma Terms of Use
  	Last modified: February 21, 2024

>>>

프롬프트로 질문 "why is the sky blue?"

>>> why is the sky blue?
The sky appears blue due to a phenomenon called **Rayleigh scattering**.

Here's a breakdown:

* **Sunlight:** Sunlight is made up of all the colors of the rainbow.
* **Atmosphere:** When sunlight enters Earth's atmosphere, it collides with tiny air molecules (mainly nitrogen and
oxygen).
* **Scattering:** These molecules scatter the sunlight in all directions.
* **Wavelengths:**  Blue light has a shorter wavelength than other colors in the visible spectrum. Shorter wavelengths are
scattered more effectively by the air molecules.

Therefore, blue light gets scattered much more than other colors, making the sky appear blue to our eyes.

**At sunrise and sunset:** The sunlight travels through more atmosphere to reach us. This means even more blue light is
scattered away, allowing longer wavelengths like orange and red to dominate, resulting in those beautiful hues.

>>> Send a message (/? for help)
Use Ctrl + d or /bye to exit.

REST 엔드포인트를 통해 응답 생성

(gabriel) Gabrielui-MacBookPro:~ gabriel$ curl http://localhost:11434/api/generate -d '{
>   "model": "gemma2",
>   "prompt": "Why is the sky blue?"
> }'
{"model":"gemma2","created_at":"2024-09-05T15:30:49.100401Z","response":"The","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:30:49.464952Z","response":" sky","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:30:49.798186Z","response":" appears","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:30:50.143292Z","response":" blue","done":false}



{"model":"gemma2","created_at":"2024-09-05T15:32:05.437344Z","response":" have","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:05.785728Z","response":" any","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:06.14108Z","response":" other","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:06.517148Z","response":" questions","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:06.926336Z","response":"!","done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:07.269795Z","response":"","done":true,"done_reason":"stop","context":[106,1645,108,4385,603,573,8203,3868,235336,107,108,106,2516,108,651,8203,8149,3868,3402,577,476,27171,3151,5231,15599,44957,38497,168428,235248,109,4858,235303,235256,476,25497,235292,109,235287,5231,219715,66058,175521,603,4247,1644,908,576,832,573,9276,576,573,30088,235265,108,235287,5231,30962,235303,235256,117961,66058,3194,33365,30866,573,10379,235303,235256,13795,235269,665,1041,23524,675,16791,2681,24582,591,80711,23584,578,16175,846,108,235287,5231,102164,574,66058,3766,24582,17109,573,33365,575,832,16759,235265,235248,108,235287,5231,10716,181430,235256,66058,7640,2611,919,476,25270,35571,1178,1156,9276,575,573,12918,18303,235265,10323,576,736,235269,665,6803,30390,1683,978,731,573,2681,24582,1178,5543,95178,1154,3118,689,10436,235265,109,688,2339,235269,1212,783,1443,66058,109,651,30390,3868,2611,26676,1167,4628,774,832,16759,235269,3547,573,8203,4824,3868,235265,2625,48825,578,22097,235269,573,33365,37845,1593,978,576,573,13795,235265,1417,3454,573,3868,2611,603,30390,3024,1693,978,235269,15267,573,5543,95178,591,165117,578,72638,235275,577,6378,1167,4628,235269,10241,1941,4964,9276,235265,110,5331,682,1230,1013,692,791,1089,1156,3920,235341],"total_duration":81320587916,"load_duration":42868270,"prompt_eval_count":15,"prompt_eval_duration":3107284000,"eval_count":210,"eval_duration":78169448000}
(gabriel) Gabrielui-MacBookPro:~ gabriel$

REST 엔드포인트를 통해 Gemma와 채팅

(gabriel) Gabrielui-MacBookPro:~ gabriel$ curl http://localhost:11434/api/chat -d '{
>   "model": "gemma2",
>   "messages": [
>     {
>       "role": "user",
>       "content": "why is the sky blue?"
>     }
>   ]
> }'
{"model":"gemma2","created_at":"2024-09-05T15:32:34.118301Z","message":{"role":"assistant","content":"The"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:34.466319Z","message":{"role":"assistant","content":" sky"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:34.821353Z","message":{"role":"assistant","content":" appears"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:35.182394Z","message":{"role":"assistant","content":" blue"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:32:35.523289Z","message":{"role":"assistant","content":" due"},"done":false}

{"model":"gemma2","created_at":"2024-09-05T15:33:50.644094Z","message":{"role":"assistant","content":" if"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:51.039595Z","message":{"role":"assistant","content":" you"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:51.411561Z","message":{"role":"assistant","content":" have"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:51.815174Z","message":{"role":"assistant","content":" any"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:52.152768Z","message":{"role":"assistant","content":" other"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:52.492949Z","message":{"role":"assistant","content":" questions"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:52.915517Z","message":{"role":"assistant","content":"!"},"done":false}
{"model":"gemma2","created_at":"2024-09-05T15:33:53.27064Z","message":{"role":"assistant","content":""},"done_reason":"stop","done":true,"total_duration":80946825634,"load_duration":44007406,"prompt_eval_count":15,"prompt_eval_duration":1749943000,"eval_count":190,"eval_duration":79152305000}
(gabriel) Gabrielui-MacBookPro:~ gabriel$

Ollama + Gemma 1.1 (2B)

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama run gemma:2b
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling manifest
pulling c1864a5eb193... 100% ▕██████████████████████████████████████████████████████████████████████████████████▏ 1.7 GB
pulling 097a36493f71... 100% ▕██████████████████████████████████████████████████████████████████████████████████▏ 8.4 KB
pulling 109037bec39c... 100% ▕██████████████████████████████████████████████████████████████████████████████████▏  136 B
pulling 22a838ceb7fb... 100% ▕██████████████████████████████████████████████████████████████████████████████████▏   84 B
pulling 887433b89a90... 100% ▕██████████████████████████████████████████████████████████████████████████████████▏  483 B
verifying sha256 digest
writing manifest
success
>>> Send a message (/? for help)

다운로드한 모델을 확인합니다.

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME         	ID          	SIZE  	MODIFIED
gemma:2b     	b50d6c999e59	1.7 GB	3 minutes ago
gemma2:latest	ff02c3702f32	5.4 GB	40 minutes ago
(gabriel) Gabrielui-MacBookPro:~ gabriel$

Gemma2와 동일한 질문을 합니다. "why is the sky blue?"

>>> why is the sky blue?
The sky appears blue due to Rayleigh scattering.

**Rayleigh scattering** is a scattering phenomenon where light waves are scattered in different directions depending on their wavelength.
Blue light has a shorter wavelength than other colors of light, meaning it scatters more effectively. This is why the sky appears blue.

Here's a more detailed explanation:

* **Sunlight** contains all colors of the rainbow.
* **Blue light** has a shorter wavelength than other colors.
* **When blue light** interacts with molecules in the atmosphere, it **scatters more strongly** than other colors.
* This scattering causes blue light to be scattered **towards the observer**.
* The scattered blue light is what we see in the sky as **blue color**.

**Other factors that contribute to the blue color of the sky:**

* **Dust particles:** Air molecules can also scatter light, but their scattering effect is generally weaker than that of molecules in the
atmosphere.
* **Water droplets:** Water droplets in the atmosphere can also scatter light, but their effect is typically less significant than that of
molecules.
* **Sun's position:** The sky appears blue because the sun is lower in the sky, meaning it is directly overhead. This means that more blue
light reaches our eyes.

In summary, the blue color of the sky is due to Rayleigh scattering of light by molecules in the atmosphere, which primarily results from
the shorter wavelength of blue light.

>>> Send a message (/? for help)

실행시에 MacBook의 성능을 확인합니다.

실행한 모델 정보 확인

>>> /show info
  Model
  	arch            	gemma
  	parameters      	3B
  	quantization    	Q4_0
  	context length  	8192
  	embedding length	2048

  Parameters
  	repeat_penalty	1
  	stop          	"<start_of_turn>"
  	stop          	"<end_of_turn>"

  License
  	Gemma Terms of Use
  	Last modified: February 21, 2024

>>> Send a message (/? for help)

Ollama + Gemma 1.1 (7B)

이번엔 Gemma 7B 모델을다운받아 테스트합니다.

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama run gemma:7b
pulling manifest
pulling ef311de6af9d...  74% ▕█████████████████████████████████████████
pulling manifest
pulling ef311de6af9d...  74% ▕█████████████████████████████████████████
pulling manifest
pulling ef311de6af9d...  74% ▕█████████████████████████████████████████
pulling manifest
pulling ef311de6af9d... 100% ▕███████████████████████████████████████████████████████████████▏ 5.0 GB
pulling 097a36493f71... 100% ▕███████████████████████████████████████████████████████████████▏ 8.4 KB
pulling 109037bec39c... 100% ▕███████████████████████████████████████████████████████████████▏  136 B
pulling 65bb16cf5983... 100% ▕███████████████████████████████████████████████████████████████▏  109 B
pulling 0c2a5137eb3c... 100% ▕███████████████████████████████████████████████████████████████▏  483 B
verifying sha256 digest
writing manifest
success
>>> Send a message (/? for help)

다운로드한 모델 확인

(gabriel) Gabrielui-MacBookPro:~ gabriel$ ollama list
NAME             ID              SIZE      MODIFIED
gemma:7b         a72c7f4d0a15    5.0 GB    2 weeks ago
gemma:2b         b50d6c999e59    1.7 GB    2 weeks ago
gemma2:latest    ff02c3702f32    5.4 GB    2 weeks ago

Gemma2와 동일한 질문을 합니다. "why is the sky blue?"

>>> why is the sky blue?
**The sky is blue due to a phenomenon called Rayleigh scattering.**

* Sunlight is composed of all the colors of the rainbow, each with a different wavelength.
* Shorter wavelengths of light, such as blue light, have higher energy and more tightly spaced photons.
* When sunlight interacts with molecules in the atmosphere, such as nitrogen and oxygen, the shorter wavelengths are
scattered more effectively.

**Here's how it works:**

1. Sunlight enters the atmosphere and interacts with molecules of nitrogen and oxygen.
2. These molecules have electrons that can be excited by the energy of the incoming light.
3. The electrons absorb the energy and vibrate, then emit it in all directions.
4. However, **blue light is scattered more efficiently than other colors** because its shorter wavelength allows for more
frequent collisions between the photons and the molecules.

**This scattering process results in:**

* More blue light is scattered in all directions, reaching our eyes and making the sky appear blue.
* Longer wavelengths of light, such as red light, are scattered less efficiently, so we see more of them directly from
the sun.

**Additional factors influencing the sky's color:**

* **Time of day:** The sky is generally bluer at midday when the sunlight travels through less atmosphere.
* **Cloud cover:** Clouds can block the sunlight and scatter less blue light, making the sky appear less blue.
* **Pollution:** Pollution in the atmosphere can also scatter light and affect the sky's color.

>>> Send a message (/? for help)

실행한 모델 정보 확인

>>> /show info
  Model
  	arch            	gemma
  	parameters      	9B
  	quantization    	Q4_0
  	context length  	8192
  	embedding length	3072

  Parameters
  	penalize_newline	false
  	repeat_penalty  	1
  	stop            	"<start_of_turn>"
  	stop            	"<end_of_turn>"

  License
  	Gemma Terms of Use
  	Last modified: February 21, 2024

>>> Send a message (/? for help)

질의 성능 확인

Intel CPU의 MacBook에서 Gemma2 9B 모델 그리고 Gemma 2B, 7B 모델까지 Ollama로 구동해 봤습니다.

감사합니다.

테디님이 올려놓으신 유튜브 영상을 보고 Macbook M1에서 따라해봤습니다.  중간에 몇가지 설정에 맞게 약간 변경해가면서 테스트 합니다.

https://www.youtube.com/watch?v=VkcaigvTrug&t=23s

# GPU 모니터링
% sudo asitop

# LangServe 실행
% python server.py

# ngrok으로 external 서비스
% ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

# PDF를 RAG
% streamlit run main.py

conda 가상환경 만들고 requirements.txt로 필요한 모듈 설치

conda create -n llm311 python=3.11

% conda env list
# conda environments:
#
base                  *  /Users/dongsik/miniconda
llm                      /Users/dongsik/miniconda/envs/llm
llm311                   /Users/dongsik/miniconda/envs/llm311

% conda activate llm311

% python -V
Python 3.11.9

% pip list
Package    Version
---------- -------
pip        23.3.1
setuptools 68.2.2
wheel      0.41.2

예제 github을 내 github으로 fork 한후 내 PC에 clone 받아서 내환경에 맞게 수정하면서 진행합니다.

teddy github : https://github.com/teddylee777/langserve_ollama

내 github : https://github.com/dongshik/langserve_ollama

% ll
total 1000
drwxr-xr-x@ 12 dongsik  staff     384 Apr 20 09:22 .
drwxr-xr-x   4 dongsik  staff     128 Apr 19 16:35 ..
drwxr-xr-x@ 14 dongsik  staff     448 Apr 19 16:40 .git
-rw-r--r--@  1 dongsik  staff      50 Apr 19 16:35 .gitignore
-rw-r--r--@  1 dongsik  staff    3343 Apr 19 16:35 README.md
drwxr-xr-x@  8 dongsik  staff     256 Apr 19 16:35 app
drwxr-xr-x@  8 dongsik  staff     256 Apr 19 16:35 example
drwxr-xr-x@  3 dongsik  staff      96 Apr 19 16:35 images
drwxr-xr-x@  4 dongsik  staff     128 Apr 19 16:35 ollama-modelfile
-rw-r--r--@  1 dongsik  staff  481043 Apr 19 16:35 poetry.lock
-rw-r--r--@  1 dongsik  staff     659 Apr 19 16:35 pyproject.toml
-rw-r--r--@  1 dongsik  staff   14983 Apr 19 16:35 requirements.txt

pip install -r requirements.txt

% pip install -r requirements.txt
Ignoring colorama: markers 'python_version >= "3.11.dev0" and python_version < "3.12.dev0" and platform_system == "Windows"' don't match your environment

% pip list | grep lang
langchain                  0.1.16
langchain-community        0.0.32
langchain-core             0.1.42
langchain-openai           0.1.3
langchain-text-splitters   0.0.1
langchainhub               0.1.15
langdetect                 1.0.9
langserve                  0.0.51
langsmith                  0.1.47

% pip list | grep huggingface
huggingface-hub            0.22.2

Huggingface에서 모델 Download 받고 Ollama에 EEVE Q5 모델 등록하고 구동

huggingface-cli download \
  heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF \
  ggml-model-Q5_K_M.gguf \
  --local-dir /Users/dongsik/GitHub/teddylee777/langserve_ollama/ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0 \
  --local-dir-use-symlinks False
  
Consider using `hf_transfer` for faster downloads. This solution comes with some limitations. See https://huggingface.co/docs/huggingface_hub/hf_transfer for more details.
downloading https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/resolve/main/ggml-model-Q5_K_M.gguf to /Users/dongsik/.cache/huggingface/hub/tmpkuuur4ki
ggml-model-Q5_K_M.gguf:  37%|███████████████████████████▌                                               | 2.81G/7.65G [04:35<09:55, 8.13MB/s]

% ll -sh
total 14954512
       0 drwxr-xr-x@ 5 dongsik  staff   160B Apr 20 10:02 .
       0 drwxr-xr-x@ 4 dongsik  staff   128B Apr 20 10:02 ..
       8 -rw-r--r--@ 1 dongsik  staff   369B Apr 19 16:35 Modelfile
       8 -rw-r--r--@ 1 dongsik  staff   419B Apr 19 16:35 Modelfile-V02
14954496 -rw-r--r--  1 dongsik  staff   7.1G Apr 20 10:02 ggml-model-Q5_K_M.gguf

<경로>/langserve_ollama/ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

FROM ggml-model-Q5_K_M.gguf

TEMPLATE """{{- if .System }}
<s>{{ .System }}</s>
{{- end }}
<s>Human:
{{ .Prompt }}</s>
<s>Assistant:
"""

SYSTEM """A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."""

PARAMETER TEMPERATURE 0
PARAMETER stop <s>
PARAMETER stop </s>

모델파일 설정하지 않으면 답변이 끝났을때 이상하게 대답할수도있기때문에 필요합니다. 

System prompt가 있다면 중간(.System) 위치에 넣어으라는 의미이며 여기서는 'SYSTEM'이 이자리를 치환하게 됩니다.

그다음 <s> 스페셜 토큰이 앞에 붙어서 사용자 즉 Human의 질문 .Prompt가 들어가게 됩니다.

그후 모델 Assistant가 받아서 답변하게 됩니다. 

※ Note!!

Modelfile에서 <s>는 문장의 시작을 나타내는 특수 토큰입니다. 이것은 "문장의 시작"을 나타내기 위해 사용됩니다. 예를 들어, 자연어 처리 작업에서 모델이 문장의 시작을 식별하고, 이에 따라 적절한 처리를 수행할 수 있도록 합니다. 이것은 토큰화된 데이터의 일부로서 모델에 제공됩니다.

tokenizer.chat_template

https://huggingface.co/yanolja/EEVE-Korean-Instruct-10.8B-v1.0

Prompt Template

A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions.
Human: {prompt}
Assistant:

ollama 목록 확인 

% ollama list
NAME    	ID          	SIZE  	MODIFIED
eeve:q4 	68f4c2c2d9fe	6.5 GB	7 days ago
gemma:2b	b50d6c999e59	1.7 GB	10 days ago

ollama가 잘 구동되어 있는지 확인합니다.

% ps -ef | grep ollama
  501  3715  3691   0 Wed01PM ??         0:29.51 /Applications/Ollama.app/Contents/Resources/ollama serve
  501  4430     1   0 Wed01PM ??         0:00.03 /Applications/Ollama.app/Contents/Frameworks/Squirrel.framework/Resources/ShipIt com.electron.ollama.ShipIt /Users/dongsik/Library/Caches/com.electron.ollama.ShipIt/ShipItState.plist
  501 61608  3197   0 10:45AM ttys002    0:00.01 grep ollama

새로받은 모델을 ollama에 등록합니다

ollama create eeve:q5 -f ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

저는 위의 Modelfile로 ollama등록할려고 하니 "Error: unknown parameter 'TEMPERATURE'"가 발생했습니다. 
그래서 소문자 temperature로 변경해서 생성되었습니다.
만일 동일한 에러가 발생한다면 소문자 temperature로 변경해서 생성해보시기 바랍니다. 

% ollama create eeve:q5 -f ollama-modelfile/EEVE-Korean-Instruct-10.8B-v1.0/Modelfile

transferring model data
creating model layer
creating template layer
creating system layer
creating parameters layer
creating config layer
using already created layer sha256:b9e3d1ad5e8aa6db09610d4051820f06a5257b7d7f0b06c00630e376abcfa4c1
writing layer sha256:6b70a2ad0d545ca50d11b293ba6f6355eff16363425c8b163289014cf19311fc
writing layer sha256:1fa69e2371b762d1882b0bd98d284f312a36c27add732016e12e52586f98a9f5
writing layer sha256:3ab8c1bbd3cd85e1b39b09f5ff9a76e64da20ef81c22ec0937cc2e7076f1a81c
writing layer sha256:d86595b443c06710a3e5ba27700c6a93ded80100ff1aa808a7f3444ff529fa70
writing manifest
success

% ollama list
NAME    	ID          	SIZE  	MODIFIED
eeve:q4 	68f4c2c2d9fe	6.5 GB	7 days ago
eeve:q5 	0732d4a47219	7.7 GB	7 minutes ago
gemma:2b	b50d6c999e59	1.7 GB	10 days ago

ollama run eeve:q5

% ollama run eeve:q5
>>> 대한민국의 수도는 어디야?
안녕하세요! 대한민국의 수도에 대해 궁금해하시는군요. 서울이 바로 그 곳입니다! 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를 가진 대도시로,
세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며, 많은 바와 클럽
관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?

>>>

아래 문구로 질문해보겠습니다.

한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주

>>> 한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주
대한민국의 수도에 대한 질문에 답변해 주셔서 감사합니다! 정답은 (D) 서울입니다. 서울은 나라의 북부에 위치해 있으며 정치, 경제, 문화의 중심지 역할을 하고 있습니다. 2019년 기준으로 약 970만 명의 인구를
대도시로, 세계에서 가장 큰 도시 중 하나입니다. 또한 세계적인 금융 허브이자 주요 관광지로, 경복궁, 남산타워, 명동과 같은 다양한 역사적 및 현대적 명소를 자랑하고 있습니다. 서울은 활기찬 밤문화로도 유명하며,
많은 바와 클럽이 관광객과 현지인 모두를 끌어들입니다. 대한민국의 수도에 대해 더 알고 싶으신 것이 있으신가요?

질문과 동시에 답변이 나오는것처럼 작동합니다. 속도도 좋고 답변의 퀄리티도 좋습니다.

>>> 다음 지문을 읽고 문제에 답하시오.
...
... ---
...
... 1950년 7월, 한국 전쟁 초기에 이승만 대통령은 맥아더 장군에게 유격대원들을 북한군의 후방에 침투시키는 방안을 제안했다. 이후, 육군본부는 육본직할 유격대와 육본 독립 유격대를 편성했다. 국군은 포항과 인접한 장사동 지역에 상륙작
... 전을 수행할 부대로 독립 제1유격대대를 선정했다. 육군본부는 독립 제1유격대대에 동해안의 장사동 해안에 상륙작전을 감행하여 북한군 제2군단의 보급로를 차단하고 국군 제1군단의 작전을 유리하게 하기 위한 작전명령(육본 작명 제174호)
... 을 하달했다. 9월 14일, 독립 제1유격대대는 부산에서 LST 문산호에 승선하여 영덕군의 장사동으로 출항했다.
...
... 1950년 9월 15일, 독립 제1유격대대는 장사동 해안에 상륙을 시도하였으나 태풍 케지아로 인한 높은 파도와 안개로 인해 어려움을 겪었다. LST 문산호는 북한군의 사격과 파도로 인해 좌초되었고, 상륙부대는 09:00시경에 전원이
... 상륙을 완료하였다. 그 후, 15:00시경에 200고지를 점령하였고, 다양한 무기와 장비를 노획하였다. 9월 16일과 17일에는 독립 제1유격대대가 여러 위치에서 북한군과의 전투를 벌였으며, 미 구축함과의 연락 두절로 인해 추가적인
... 어려움을 겪었다.
...
... 장사동에서 위급한 상황에 처한 독립 제1유격대대를 구출하기 위해 해군본부는 LT-1(인왕호)를 급파했으나, LST 문산호의 구출에 실패했다. 해군본부는 상륙부대의 철수를 지원하기 위해 LST 조치원호를 현지로 보냈다. 9월 18일,
... 이명흠 부대장은 유엔 해군과의 협력 하에 부족한 식량과 탄약 지원을 받았다. 9월 19일, 유엔군의 함포지원과 함께 LST 조치원호가 도착하여 철수 작전을 시작했다. 스피어 소령은 직접 해안에 상륙하여 구조작전을 지시하였다. 9월 2
... 0일, 725명이 부산항으로 복귀했으나, 32명이 장사동 해안에 남아 북한군의 포로가 되었거나 탈출하여 국군에 합류하였다.
...
... 장사리 전투가 인천 상륙작전의 양동작전으로 알려졌으나, 이 전투가 드라마틱한 요소로 인해 과장되었으며, 실제로는 인천 상륙작전과 큰 관련이 없다. 또한, 북한이나 중국의 군사적 상황을 고려할 때, 장사리에서의 전투가 낙동강 전선에 영
... 향을 끼칠 가능성은 낮다.
...
... ---
...
... 문제
... 1. 지문에 나오는 지명을 모두 쓰시오.
... 2. 그중 대게로 유명한 곳은?
지문에 나오는 지명은 다음과 같습니다:
- 포항
- 장사동
- 영덕군
- 부산
- 문산호
- 조치원호
- 스피어 소령
- 낙동강 전선
대게로 유명한 곳은 영덕군입니다.

ollama 쉘에서 나올때는 Use Ctrl + d or /bye to exit.

2021년형 14인치 MacBook Pro의 M1 Pro CPU는 10코어이고 GPU는 16코어입니다. 

2020년형 13인치 MacBook Pro의 M1 CPU는 8코어이고 GPU는 8코어입니다. (저는 이겁니다)

cpu를 100% 까지 사용하면서 일했습니다. (수고했어)

LangServe로 모델 서빙

langserve_ollama % ll app
total 40
drwxr-xr-x@  8 dongsik  staff   256 Apr 19 16:35 .
drwxr-xr-x@ 12 dongsik  staff   384 Apr 20 09:22 ..
-rw-r--r--@  1 dongsik  staff     0 Apr 19 16:35 __init__.py
-rw-r--r--@  1 dongsik  staff   549 Apr 19 16:35 chain.py
-rw-r--r--@  1 dongsik  staff   723 Apr 19 16:35 chat.py
-rw-r--r--@  1 dongsik  staff   328 Apr 19 16:35 llm.py
-rw-r--r--@  1 dongsik  staff  1444 Apr 19 16:35 server.py
-rw-r--r--@  1 dongsik  staff   559 Apr 19 16:35 translator.py
(llm311) dongsik@dongsikleeui-MacBookPro langserve_ollama %

chat.py, chain.py, llm.py, translator.py 세개 파일의 llm 모델명을 내 환경에 맞게 수정합니다.

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
#llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

server.py 실행

(llm311) dongsik@dongsikleeui-MacBookPro langserve_ollama % cd app
(llm311) dongsik@dongsikleeui-MacBookPro app % pwd
/Users/dongsik/GitHub/teddylee777/langserve_ollama/app
(llm311) dongsik@dongsikleeui-MacBookPro app % ll
total 40
drwxr-xr-x@  8 dongsik  staff   256 Apr 19 16:35 .
drwxr-xr-x@ 12 dongsik  staff   384 Apr 20 09:22 ..
-rw-r--r--@  1 dongsik  staff     0 Apr 19 16:35 __init__.py
-rw-r--r--@  1 dongsik  staff   584 Apr 20 13:15 chain.py
-rw-r--r--@  1 dongsik  staff   758 Apr 20 13:15 chat.py
-rw-r--r--@  1 dongsik  staff   363 Apr 20 13:15 llm.py
-rw-r--r--@  1 dongsik  staff  1444 Apr 19 16:35 server.py
-rw-r--r--@  1 dongsik  staff   594 Apr 20 13:15 translator.py
(llm311) dongsik@dongsikleeui-MacBookPro app % python server.py

http://0.0.0.0:8000/prompt/playground/

질문 과 답변

RemoteRunable로 LangServe를 호출 하도록 변경

<경로>/langserve_ollama/example

% ll
total 120
drwxr-xr-x@  9 dongsik  staff    288 Apr 20 13:50 .
drwxr-xr-x@ 12 dongsik  staff    384 Apr 20 09:22 ..
drwxr-xr-x@  3 dongsik  staff     96 Apr 19 16:35 .streamlit
-rw-r--r--@  1 dongsik  staff  12504 Apr 19 16:35 00-ollama-test.ipynb
-rw-r--r--@  1 dongsik  staff   4885 Apr 19 16:35 01-remote-invoke.ipynb
-rw-r--r--@  1 dongsik  staff   3775 Apr 19 16:35 02-more-examples.ipynb
-rw-r--r--@  1 dongsik  staff   6222 Apr 19 16:35 main.py
-rw-r--r--@  1 dongsik  staff  14708 Apr 19 16:35 requirements.txt

01-remote-invoke.ipynb의 로컬 LangServe 주소로 변경합니다

from langserve import RemoteRunnable

# ngrok remote 주소 설정

#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
# chain = RemoteRunnable("https://poodle-deep-marmot.ngrok-free.app/prompt/")
chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")

for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
    print(token, end="")

ngrok을 이용해서 로컬 LangServe 를 Port Forwarding하기 

ngrok 가입 

https://dashboard.ngrok.com/cloud-edge/domains

M1용 설치 파일을 다운로드 받아서 설치합니다.

https://dashboard.ngrok.com/get-started/setup/macos

무료 도메인 설정 

humble-curiously-antelope.ngrok-free.app

LangServe 구동된 포트로 ngok 도메인 지정해서 포트 포워딩

ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

% ngrok http --domain=humble-curiously-antelope.ngrok-free.app 8000

ngrok                                                                                                                    (Ctrl+C to quit)

K8s Gateway API support available now: https://ngrok.com/r/k8sgb

Session Status                online
Account                       dongsik.lee (Plan: Free)
Version                       3.8.0
Region                        Japan (jp)
Latency                       45ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    https://humble-curiously-antelope.ngrok-free.app -> http://localhost:8000

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

https://humble-curiously-antelope.ngrok-free.app/prompt/playground/

ngrok url로 질의를 해보면 local 서버의 GPU가 100%로 올라가면서 Output을 만들고있습니다.

01-remote-invoke.ipynb 파일의 RemoteRunnable 주소를 ngrok 주소로 변경하고 vscode로 실행해봅니다.

from langserve import RemoteRunnable

# ngrok remote 주소 설정

#chain = RemoteRunnable("NGROK 에서 설정한 본인의 도메인 주소/prompt/")
chain = RemoteRunnable("https://humble-curiously-antelope.ngrok-free.app/prompt/")
#chain = RemoteRunnable("http://0.0.0.0:8000/prompt/")

for token in chain.stream({"topic": "딥러닝에 대해서 알려줘"}):
    print(token, end="")

잘 작동됩니다.

추가 예제 

번역기 

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

# 프롬프트 설정
prompt = ChatPromptTemplate.from_template(
    "Translate following sentences into Korean:\n{input}"
)

# LangChain 표현식 언어 체인 구문을 사용합니다.
chain = prompt | llm | StrOutputParser()

LLM을 Runable로 실행

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate

# LangChain이 지원하는 다른 채팅 모델을 사용합니다. 여기서는 Ollama를 사용합니다.
# llm = ChatOllama(model="EEVE-Korean-10.8B:latest")
llm = ChatOllama(model="eeve:q5")

Streamlit으로 PDF rag 해보기

Embedding을 OpenAIEmbeddings을 사용하기위해서 OPENAI_API_KEY를 .env 파일에서 가져옵니다.

% pip install python-dotenv

main.py 내용중 OPEN API KEY세팅과 LANGSERVE_ENDPOINT를 ngrok주소로 업데이트 한후 실행합니다

% streamlit run main.py

  You can now view your Streamlit app in your browser.

  Local URL: http://localhost:8501
  Network URL: http://192.168.0.10:8501

  For better performance, install the Watchdog module:

  $ xcode-select --install
  $ pip install watchdog

예제 > SPRI_AI_Brief_2023년12월호_F.pdf

https://spri.kr/posts?code=AI-Brief

FileNotFoundError: [Errno 2] No such file or directory: 'pdfinfo'

% conda install poppler
Channels:
 - defaults
 - conda-forge
Platform: osx-arm64

% pip install pdftotext

FileNotFoundError: [Errno 2] No such file or directory: 'tesseract'

% brew install tesseract
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

% brew install tesseract-lang

UnicodeEncodeError: 'ascii' codec can't encode characters in position 22-23: ordinal not in range(128)

위 PDF에서 최종 질문 을 해보겠습니다.

실제해보니 내용이 엄청난 영상입니다. 

- Ollama

- EEVE 양자화 모델

- LangServe

- ngrok

- Streamlit RAG

- Asitop

감사합니다.

Asitop으로 내 M1 상태 모니터링

% pip install asitop

% sudo asitop 

sudo 패스워드 입력

앞에서 Ollama로 Gemma 경량화 모델을 실행해봤습니다. 이번엔 한글을 지원하는 경량화 모델중에 

https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF

사이트에 설치 방법과 테스트 방법이 잘 기술되어있습니다. 다만, GPU Cuda 버전이 안맞을경우 가이드대로 실행할때 Execption이 발생해서

• Original model is yanolja/EEVE-Korean-Instruct-10.8B-v1.0
• quantized using llama.cpp

llama.cpp는 4비트 정수 양자화를 이용해서 Llama 모델과 Python이 함께 실행하는(저수준 액세스 바인더) 것을 목표로 만들어진 프로젝트입니다. 의존성 없는 순수 C/C++를 통해서 구현되었으며, Mac OS, Windows, Linux 모두 실행 가능합니다.

다운로드 가능한 세가지 모델의 비교입니다

셋중에서 가장작은 Q4 모델을 다운로드 받습니다.

https://huggingface.co/heegyu/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/resolve/main/ggml-model-Q4_K_M.gguf?download=true

그리고 modelfile 파일을 만들어서 다운로드 받은 모델을 Ollama에 등록해주면 됩니다.

다운로드 받는 GGML은 Apple M1 및 M2 실리콘에 최적화된 양자화 구현이라고 합니다.

다운로드 받은 모델파일을 Ollama에 등록해주기 위해서 Modelfile을 생성합니다.

ModelFile

FROM /Users/dongsik/GitHub/llm/eeve/EEVE-Korean-Instruct-10.8B-v1.0-GGUF/ggml-model-Q4_K_M.gguf
​
TEMPLATE """### User:
{{ .Prompt }}
​
### Assistant:
"""
​
PARAMETER temperature 0.1
​
PARAMETER num_ctx 4096
PARAMETER stop "</s>"
PARAMETER stop "### System:"
PARAMETER stop "### User:"
PARAMETER stop "### Assistant:"

Modelfile과 다운로드받은 GGUF 파일이 준비되었습니다.

% ll
total 40
drwxr-xr-x  8 dongsik  staff   256 Apr 12 14:28 .
drwxr-xr-x  7 dongsik  staff   224 Apr 12 11:33 ..
drwxr-xr-x  5 dongsik  staff   160 Apr 11 00:11 .ipynb_checkpoints
drwxr-xr-x  3 dongsik  staff    96 Apr 10 23:55 EEVE-Korean-Instruct-10.8B-v1.0-GGUF
-rw-r--r--  1 dongsik  staff   325 Apr 12 14:27 Modelfile
-rw-r--r--  1 dongsik  staff  5957 Apr 11 00:26 ollama_eeve_gguf.ipynb

eeve 모델을 Ollama에 등록해줍니다

% ollama create eeve:q4 -f Modelfile
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
2024/04/12 14:28:50 parser.go:73: WARN Unknown command:
transferring model data
creating model layer
creating template layer
creating parameters layer
creating config layer
using already created layer sha256:5a79b80eb5e2eec5cf5d514dfa32187872dde1dae6a2b9c8
using already created layer sha256:c3de887d2d041bfea1bfed395834ea828839af278003269e
using already created layer sha256:e6b785eab1777ecfc57eab9a85f9b623931e6f1079ae6d75
using already created layer sha256:8b03799cdb5862e5cdfda70f0e116193aa07f2309015a158
writing manifest
success

성공적으로 등록되면 모델을 확인가능합니다.

gemma:2b 모델과 eeve:q4 모델 두개가 등록된것을 확인합니다.

% ollama list
NAME       	ID          	SIZE  	MODIFIED
eeve:q4    	68f4c2c2d9fe	6.5 GB	8 seconds ago
gemma:2b   	b50d6c999e59	1.7 GB	2 days ago

지울때는 rm 명령을 사용합니다 (ollama rm eeve:q4)

cli로 ollama eeve:q4 모델에 질문을 해봅니다. 

M1 노트북임을 감안했을때 상당히 빠르게 답변을 해주고 있고 퀄리티도 상당합니다.

아래 추가로 ChatGPT 3.5와 뤼튼의 GPT 4를 통해서 동일한 질문을 해본 결과 입니다.

eeve:q4 의 답변

% ollama run eeve:q4 하늘은 왜 파란색일까요?
하늘이 파란색인 이유는 대기 중에 있는 공기 분자와 먼지 입자들이 햇빛을 산란시키기 때문입니다. 태양빛이 지구 대기를 통과할 때, 이 작은 입자들과 상호 작용하여 빛을 다양한 방향으로 흩어지게 합니다. 우리가 하늘에서 보는 빛은 주로
파란색과 보라색의 가시광선 스펙트럼에 집중되어 있습니다.

이 현상은 빛이 공기 분자와 먼지 입자에 부딪힐 때 일어나는 산란 효과 때문입니다. 이 작은 입자들은 태양빛을 다양한 방향으로 흩어지게 하여, 우리가 하늘에서 보는 색깔을 만들어냅니다. 파란색과 보라색 빛은 다른 색들보다 더
산란되기 때문에, 우리 눈에 가장 많이 도달하여 하늘을 파랗게 보이게 합니다.

또한 대기 중에 있는 물방울이나 얼음 결정도 햇빛의 산란에 역할을 할 수 있습니다. 이 입자들이 태양빛과 상호 작용할 때, 빛을 다양한 방향으로 흩어지게 하여 하늘을 더욱 파랗게 보이게 만듭니다.

하늘이 항상 파란색인 것은 아니며, 대기 조건에 따라 색깔이 변할 수 있다는 점을 주목하는 것이 중요합니다. 예를 들어, 일출이나 일몰 때는 대기 중에 있는 먼지 입자와 물방울의 양이 증가하여 햇빛을 더 많이 산란시켜 하늘에 주황색,
분홍색 또는 보라색을 만들 수 있습니다.

요약하자면, 하늘이 파란색인 이유는 공기 분자, 먼지 입자 및 대기 중 물방울이나 얼음 결정과 같은 다양한 요소가 태양빛의 산란에 기여하기 때문입니다. 이 상호 작용은 파란색과 보라색 빛의 높은 농도를 만들어내어 우리가 하늘에서 보는
색깔을 만들어냅니다.

GPT-3.5의 답변

뤼튼 GPT-4의 답변

Ollama Langchain을 이용해서 추론을 잘하는지 질문을 해보겠습니다.

import time
import langchain
from langchain_community.llms import Ollama
import logging

# Configure basic logging
logging.basicConfig(level=logging.INFO)

try:
    llm = Ollama(model="eeve:q4")
    
    # 프롬프트가 잘 정의되어 있는지 확인하는 것이 필요합니다. (모델의 기능에 따라 조정가능)
    start = time.time()
    prompt = ("한국의 수도는 어디인가요? 아래 선택지 중 골라주세요.\n\n(A) 경성\n(B) 부산\n(C) 평양\n(D) 서울\n(E) 전주")
    
    response = llm.invoke(prompt)
    print(response)
    print(time.time() - start)
    
except ImportError:
    logging.error("Failed to import Ollama from langchain_community. Is the package installed?")
except Exception as e:
    logging.error(f"An unexpected error occurred: {e}")

정답은 (D) 서울입니다.
3.465108871459961

해당 내용은 Mac (M1) 을 기준으로 작성되었습니다.

대규모 언어 모델을 로컬에서 실행하기 위해서 Ollama를 설치하고 구글의 최첨단 경량 오픈모델인 gemma를 다운받아서 간단히 돌려보고

LangChain으로 연결해봅니다.

간밤에 "LLM RAG Langchain 통합" 채팅방의 권진영님께서 친절하게 설치와 사용방법을 알려주셔서 다른분들도 간단히 설치해서 사용해보면 좋을거 같아서 정리해봅니다.

Ollama github에 가면 로컬환경에 설치가능한 설치파일들을 다운받을수있습니다.

https://github.com/ollama/ollama

이중 macOS에 해당하는 설치파일을 다운로드 받아서 설치합니다. 설치는 너무 간단해서 의외이기도 합니다. 다운로드 받으면 디렉토리에 설치파일이 생기게되고 더블클릭해서 설치합니다.

실행하면 아래처럼 설치가 시작됩니다. Next 클릭

설치는 금방 완료됩니다.

설치가 완료되면 Terminal을 실행해서 Ollama를 실행합니다.

Gemma는 두개 모델을 제공하는데 먼저 가장작은 모델로 시작해 보겠습니다.

% ollama run gemma:2b
pulling manifest
pulling c1864a5eb193...   5% ▕███                     ▏  87 MB/1.7 GB  8.0 MB/s   3m17s

설치가 완료되면 메시지를 호출 할수있는 창이 뜨면서 설치가 완료됩니다.

% ollama run gemma:2b
pulling manifest
pulling c1864a5eb193... 100% ▕██████████████████████████████████████████████▏ 1.7 GB
pulling 097a36493f71... 100% ▕██████████████████████████████████████████████▏ 8.4 KB
pulling 109037bec39c... 100% ▕██████████████████████████████████████████████▏  136 B
pulling 22a838ceb7fb... 100% ▕██████████████████████████████████████████████▏   84 B
pulling 887433b89a90... 100% ▕██████████████████████████████████████████████▏  483 B
verifying sha256 digest
writing manifest
removing any unused layers
success
>>> hi
Hi! 👋 How can I assist you today? 😊

Is there anything I can help you with?

>>> Send a message (/? for help)

간단하게 "hi"로 인사해 봤습니다.

이제 실행창에서 나오도록 합니다 "/bye"를 입력합니다.

>>> /bye
Hello! 👋 It's nice to hear from you. How can I help you today? 😊

Is there anything I can do for you?

>>> /bye
(base) dongsik@dongsikleeui-MacBookPro ~ %

바로 내보내주지는 않으면 한번더 "/bye" 합니다

설치되어있는 모델을 확인할수도 있습니다.

% ollama list
NAME    	ID          	SIZE  	MODIFIED
gemma:2b	b50d6c999e59	1.7 GB	6 minutes ago

또한 설치된 모델을 삭제도 할수있습니다. 설치와 삭제가 너무간단합니다.

% ollama rm gemma:2b
deleted 'gemma:2b'
% ollama list
NAME	ID	SIZE	MODIFIED
%

그럼 다시 설치하고 간단히 사용방법을 설명합니다.

1. LangChain 으로 실행하기

# LangChain 설치
pip install langchain

import langchain
# LangChain 버전 확인
print('LangChain version:', langchain.__version__)

결과
LangChain version: 0.1.12

로컬에 설치든 ollama gemma:2b 모델을 사용하도록 설정하고 실행합니다.

from langchain_community.llms import Ollama
import logging

# logging 설정
logging.basicConfig(level=logging.INFO)

try:
    llm = Ollama(model="gemma:2b")
    
    # 프롬프트가 잘 정의되어 있는지 확인하는 것이 필요합니다. (모델의 기능에 따라 조정가능)
    prompt = ("Why is the sky blue?")
    
    response = llm.invoke(prompt)
    print(response)
except ImportError:
    logging.error("Failed to import Ollama from langchain_community. Is the package installed?")
except Exception as e:
    logging.error(f"An unexpected error occurred: {e}")

결과 :
The sky appears blue due to Rayleigh scattering. This scattering process occurs when light interacts with molecules in the Earth's atmosphere. 

* **Blue light has a longer wavelength than other colors**. This means it can penetrate further into the atmosphere. 
* **Blue light waves have more energy** than other colors, so they are more likely to scatter. 
* **Water vapor molecules** in the atmosphere absorb blue light more efficiently than other colors. 
* **Scattered blue light** is scattered in all directions equally, giving the sky its blue color.

The amount and intensity of blue scattering depends on several factors, including:

* **Particle size and density of the particles**: Smaller particles scatter light more efficiently than larger particles. 
* **The wavelength of light**: Blue light is scattered more strongly than other colors. 
* **Atmospheric conditions**: Temperature, humidity, and air density can also affect scattering.

Overall, the scattering of sunlight in the atmosphere creates the blue color of the sky.

url call로 호출하고 결과를 streaming 방식으로 stand out으로 출력합니다.

from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_community.llms.ollama import Ollama

llm = Ollama(
    base_url="http://localhost:11434",
    model="gemma:2b",
    callback_manager=CallbackManager(
        [StreamingStdOutCallbackHandler()],
    ),
)

prompt = ("Why is the sky blue?")
response = llm.invoke(prompt)
print(response)

결과 :
The sky appears blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength, such as blue light. This scattering causes longer wavelengths, such as red and yellow light, to be scattered more than blue light. As a result, the sky appears blue to us.The sky appears blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength, such as blue light. This scattering causes longer wavelengths, such as red and yellow light, to be scattered more than blue light. As a result, the sky appears blue to us.

https://github.com/ollama/ollama/blob/main/docs/api.md

2. Command 창에서 curl로 /api/generate

streaming 

Reqeust
% curl http://localhost:11434/api/generate -d '{
  "model": "gemma:2b",
  "prompt":"Why is the sky blue?"
}'

Response
{"model":"gemma:2b","created_at":"2024-04-10T01:46:35.254492Z","response":"The","done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:46:35.291573Z","response":" sky","done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:46:35.325664Z","response":" appears","done":false}
... <생략>
{"model":"gemma:2b","created_at":"2024-04-10T01:46:44.741546Z","response":" the","done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:46:44.775088Z","response":" atmosphere","done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:46:44.810226Z","response":".","done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:46:44.845784Z","response":"","done":true,"context":[106,1645,108,4385,603,573,8203,3868,235336,107,108,106,2516,108,651,8203,8149,3868,3402,577,153902,38497,235265,1417,38497,12702,1185,33365,113211,675,24582,575,573,10379,235303,235256,13795,235269,14076,24582,576,23584,578,16175,235265,109,235287,5231,5200,2611,919,476,5543,35571,688,1178,3868,2611,235265,1417,3454,674,1185,33365,30866,573,13795,235269,978,3868,2611,603,30390,3024,774,1167,2116,235265,108,235287,5231,10716,2611,919,476,25270,35571,688,578,603,30390,978,16347,1178,3118,2611,235265,108,235287,714,5231,10526,576,38497,688,12014,611,573,35571,576,573,2611,235265,11569,235269,3868,2611,603,30390,978,1178,1156,9276,235265,108,235287,714,13795,919,978,23584,24582,1178,16175,24582,235269,948,3454,674,978,3118,2611,603,30390,3024,235265,1417,603,3165,573,8203,8149,3868,235265,109,4858,708,1009,5942,4691,1105,573,3868,8203,235292,109,235287,714,3868,2881,603,5231,38131,576,5809,168428,1417,3454,674,573,8203,877,4824,3868,20853,576,1368,5342,689,7033,665,603,235265,108,235287,714,3868,2881,603,1170,5231,1665,10918,731,38636,168428,1417,3454,674,573,8203,877,4824,3868,793,4391,1368,1536,692,708,575,573,2134,235265,108,235287,714,3868,2881,603,476,5231,2667,576,2611,38497,168428,1417,3454,674,2611,603,30390,575,832,16759,731,24582,575,573,13795,235265,108,235287,714,3868,2881,576,573,8203,603,476,5231,28205,44299,168428,1417,603,1861,573,13795,603,780,13596,12876,235269,578,573,38497,2185,12014,611,573,6581,576,573,33365,8761,577,573,16071,575,573,13795,235265,107,108],"total_duration":13164771833,"load_duration":3454744833,"prompt_eval_count":15,"prompt_eval_duration":115430000,"eval_count":282,"eval_duration":9592944000}

No streaming 

Request
% curl http://localhost:11434/api/generate -d '{
  "model": "gemma:2b",
  "prompt":"Why is the sky blue?",
  "stream": false
}'

Reponse
{"model":"gemma:2b","created_at":"2024-04-10T01:49:38.296228Z","response":"The sky is blue due to Rayleigh scattering. Rayleigh scattering is the scattering of light by particles of a shorter wavelength. This means that blue light has a greater wavelength and is scattered more than other colors. This is why the sky appears blue.","done":true,"context":[106,1645,108,4385,603,573,8203,3868,235336,107,108,106,2516,108,651,8203,603,3868,3402,577,153902,38497,235265,153902,38497,603,573,38497,576,2611,731,16071,576,476,25270,35571,235265,1417,3454,674,3868,2611,919,476,6561,35571,578,603,30390,978,1178,1156,9276,235265,1417,603,3165,573,8203,8149,3868,235265,107,108],"total_duration":1936533375,"load_duration":3180292,"prompt_eval_duration":272404000,"eval_count":49,"eval_duration":1658092000}

3. Command 창에서 curl로 /api/chat 

Chat Request (Streaming)

Request
% curl http://localhost:11434/api/chat -d '{
  "model": "gemma:2b",
  "messages": [
    {
      "role": "user",
      "content": "why is the sky blue?"
    }
  ]
}'

Response
{"model":"gemma:2b","created_at":"2024-04-10T01:27:16.070998Z","message":{"role":"assistant","content":"The"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:16.108371Z","message":{"role":"assistant","content":" sky"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:16.142158Z","message":{"role":"assistant","content":" appears"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:16.175229Z","message":{"role":"assistant","content":" blue"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:16.207642Z","message":{"role":"assistant","content":" due"},"done":false}
... <생략>
{"model":"gemma:2b","created_at":"2024-04-10T01:27:25.624649Z","message":{"role":"assistant","content":" higher"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:25.658043Z","message":{"role":"assistant","content":" temperatures"},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:25.692536Z","message":{"role":"assistant","content":"."},"done":false}
{"model":"gemma:2b","created_at":"2024-04-10T01:27:25.725932Z","message":{"role":"assistant","content":""},"done":true,"total_duration":12244362334,"load_duration":2484924542,"prompt_eval_count":15,"prompt_eval_duration":103298000,"eval_count":286,"eval_duration":9654690000}

Chat request (No streaming)

Request
% curl http://localhost:11434/api/chat -d '{
  "model": "gemma:2b",
  "messages": [
    {
      "role": "user",
      "content": "why is the sky blue?"
    }
  ],
  "stream": false
}'

Response
{"model":"gemma:2b","created_at":"2024-04-10T01:30:16.821415Z","message":{"role":"assistant","content":"The sky appears blue due to Rayleigh scattering. This phenomenon occurs when sunlight interacts with molecules in the Earth's atmosphere.\n\n**Rayleigh Scattering:**\n\n* Sunlight is composed of all colors of the spectrum, including blue, violet, yellow, orange, and red.\n* When sunlight enters the atmosphere, it interacts with molecules such as nitrogen and oxygen molecules.\n* These molecules have different sizes and structures, which cause different wavelengths of light to scatter in different directions.\n* Blue light, with its shorter wavelengths, is scattered more strongly than other colors due to its shorter path length through the atmosphere.\n\n**Blue Sky:**\n\n* As a result, blue light is scattered in all directions from the Sun.\n* This scattering effect spreads out the Sun's light throughout the atmosphere, making the sky appear blue.\n* The intensity of blue light can vary slightly depending on factors such as altitude, temperature, and atmospheric conditions.\n\n**Other Factors:**\n\n* The scattering process depends on the size and density of the molecules, which is why the sky appears blue even though the Sun is a star of much greater temperature.\n* The atmosphere is composed of different gases with varying densities, which influences the scattering process.\n* Cloud and pollution can also affect the sky's color, with clouds reflecting blue light more efficiently than other colors.\n\n**Conclusion:**\n\nThe blue color of the sky is primarily caused by Rayleigh scattering of sunlight by molecules in the Earth's atmosphere. This scattering process spreads out the Sun's light throughout the sky, making it appear blue to us on Earth."},"done":true,"total_duration":11417865583,"load_duration":4883667,"prompt_eval_duration":270041000,"eval_count":324,"eval_duration":11140497000}