Can AI Be Your Paralegal? (Only if You Follow This 5-Step Verification Process)

 

Blogging_CS · Sep 20, 2025 · 10 min read

Generative AI promises to revolutionize the speed of legal research, but a critical pitfall lies hidden beneath the surface: “AI hallucinations.” Because AI can fabricate non-existent case law that looks authentic, legal professionals are now facing the paradox of spending more time verifying AI outputs than it would have taken to draft the work themselves.

This isn’t a hypothetical concern. In Mata v. Avianca, a case in the Southern District of New York, attorneys faced sanctions for submitting a brief containing fake judicial opinions generated by AI. Even more striking is Noland v. Land, where the California Court of Appeal sanctioned an attorney for filing a brief in which 21 of 23 case citations were complete fabrications. The penalty was severe: a $10,000 fine, mandatory notification to the client, and a report to the state bar.

These rulings send a clear message: before any discussion of technology, the user’s attitude and responsibility are paramount. Attorneys (including patent attorneys) have a fundamental, non-delegable duty to read and verify every citation in documents submitted to the court, regardless of the source. With the risk of AI hallucinations now widely known, claiming ignorance—“I didn’t know the AI could make things up”—is no longer a viable excuse. Ultimately, the final line of defense is a mindset of professional skepticism: question every AI output and cross-reference every legal basis with its original source.

---

A 5-Step Practical Workflow for Risk Management

Apply the following five-step workflow to all AI-assisted tasks to systematically manage risk.

1. Step 1: Define the Task & Select Trusted Data
   

   Set a clear objective for the AI and personally select the most reliable source materials (e.g., recent case law, statutes, internal documents). Remember that the “Garbage In, Garbage Out” principle applies from the very beginning.

2. Step 2: Draft with RAG (Retrieval-Augmented Generation)
   

   Generate the initial draft based on your selected materials. RAG is the most effective anti-hallucination technique, as it forces the AI to base its answers on a trusted external data source you provide, rather than its vast, internal training data.

   Use Case:

   • Drafting an Initial Case Memo: Upload relevant case law, articles, and factual documents to a tool like Google's NotebookLM or Claude. Then, instruct it: “Using only the uploaded documents, summarize the court's criteria for ‘Issue A’ and outline the arguments favorable to our case.” This allows for the rapid creation of a reliable initial memo.

3. Step 3: Expand Research with Citation-Enabled Tools
   

   To strengthen or challenge the initial draft's logic, use AI tools that provide source links to broaden your perspective.

   Recommended Tools:

   • Perplexity, Skywork AI: Useful for initial research as they provide source links alongside answers.
   • Gemini's Deep Research feature: Capable of comprehensive analysis on complex legal issues with citations.

   Pitfall:

   • Source Unreliability: The AI may link to personal blogs or irrelevant content. An AI-provided citation is not a verified fact; it must be checked manually.

4. Step 4: Cross-Verify with Multiple AIs & Refine with Advanced Prompts
   

   Critically review the output by posing the same question to two or more AIs (e.g., ChatGPT, Gemini, Claude) and enhance the quality of the results through sophisticated prompt engineering.

   Key Prompting Techniques:

   • Assign a Role: “You are a U.S. patent attorney with 15 years of experience specializing in the semiconductor field.”
   • Demand Chain-of-Thought Reasoning: “Think step-by-step to reach your conclusion.”
   • Instruct it to Admit Ignorance: “If you do not know the answer, state that you could not find the information rather than guessing.”

5. Step 5: Final Human Verification - The Most Critical Step
   

   You must personally check every sentence, every citation, and every legal argument generated by the AI against its original source. To skip this step is to abdicate your professional duty.

---

Advanced Strategies & Firm-Level Policy

Beyond the daily workflow, firms should establish a policy framework to ensure stability and trust in their use of AI.

• Establish a Multi-Layered Defense Framework: Consider a formal defense-in-depth approach: (Base Layer) Sophisticated prompts → (Structural Layer) RAG for grounding → (Behavioral Layer) Fine-tuning for specialization. Fine-tuning, using tools like ChatGPT's GPTs or Gemini for Enterprise, can train an AI on your firm's past work to enhance accuracy for specific tasks, but requires careful consideration of cost, overfitting, and confidentiality risks.
• Implement a Confidence-Based Escalation System: Design an internal system that scores the AI's confidence in its responses. If a score falls below a set threshold (e.g., 85%), the output could be automatically flagged for mandatory human review, creating a secondary safety net.
• Establish Principles for Billing and Client Notification: AI subscription fees should be treated as overhead, not directly billed to clients. Bill for the professional value created by using AI (e.g., deeper analysis, better strategy), not for the “machine’s time.” Include a general disclosure clause in engagement letters stating that the firm may use secure AI tools to improve efficiency, thereby ensuring transparency with clients.

---

Conclusion: Final Accountability and the Path Forward

The core of the AI hallucination problem ultimately lies in the professional’s verification mindset. The technologies and workflows discussed today are merely tools. As courts and bar associations have repeatedly warned, the final responsibility rests with the human professional.

“AI is a tool; accountability remains human.”

Only by establishing this principle and combining multi-layered verification strategies with a commitment to direct validation can we use AI safely and effectively. When we invest the time saved by AI into deeper legal analysis and more creative strategy, we evolve into true legal experts of the AI era. AI will not replace you, but the responsibility for documents bearing your name rests solely with you.

Frequently Asked Questions

Q: Can I trust the content if the AI provides a source link?

A: Absolutely not. A source link provided by an AI is merely a claim of where it got the information, not a guarantee of accuracy. The AI can misinterpret or distort the source's content. You must click the link, read the original text, and verify that it has been cited correctly and in context.

Q: What is the safest way to use AI with confidential client information?

A: The default should be to use an enterprise-grade, secure AI service contracted by your firm or a private, on-premise LLM. If you must use a public AI, you are required to completely anonymize all identifying information from your queries. Uploading sensitive data to a public AI service is a serious ethical and security violation.

Q: What is the most common mistake legal professionals make when using AI?

A: Skipping Step 5 of the workflow: “Final Human Verification.” Seeing a well-written, plausible-sounding sentence and copy-pasting it without checking the original source is the easiest way to fall into the hallucination trap, with potentially severe consequences.

Patentability of LLM Prompts: Overcoming Abstract Idea Rejections

 

Can a Simple Command to an AI Be Patented? This article provides an in-depth analysis of how LLM prompt techniques can transcend mere ‘ideas’ to be recognized as concrete ‘technical inventions,’ exploring key strategies and legal standards across different countries.

It seems that almost no one around us thinks of protecting prompt techniques or prompts that instruct LLM models with patents. At first, I was also skeptical, wondering, ‘Can a simple command to a computer be patented?’ However, as I delved deeper into this topic, I came to the conclusion that it is entirely possible if certain conditions are met. This article is a summary of the thought process I went through, and please bear in mind that it may not yet be an academically established view. 😊

 

🤔 Prompts Aren’t Patentable Because They’re Just ‘Human Thoughts,’ Right?

The first hurdle that comes to mind for many is the principle that ‘human mental processes’ are not patentable subject matter. In fact, the argument that “a prompt is fundamentally human involvement, and technology involving such human mental activity is not patentable” is one of the strongest reasons for rejection in patent examination. This standard has been particularly firm since the U.S. Supreme Court’s Alice Corp. v. CLS Bank decision. It means that merely implementing something on a computer that a person could do in their head is not enough to get a patent.

According to this logic, the act of instructing an AI through a prompt is ultimately an expression of human thought, so one could easily conclude that it cannot be patented. However, this argument is half right and half wrong. And this is precisely where our patent strategy begins.

💡 Good to Know!
What patent law takes issue with as ‘human intervention’ is not the act of giving a command to a system itself. It refers to cases where the core idea of the invention remains at the level of a mental step that can be practically performed in the human mind. Therefore, the key is to prove that our prompt technology transcends this boundary.

 

📊 A Shift in Perspective: From ‘Command’ to ‘Computer Control Technology’

The first step to unlocking the patentability of prompt technology is to change our perspective. We need to redefine our technology not as ‘a message sent from a human to an AI,’ but as ’a technology that controls the internal computational processes of a complex computer system (LLM) through structured data to solve a technical problem and achieve concrete performance improvements.’

If you take a close look at the algorithm of China’s DeepSeek-R1, you can see that it implements various prompt techniques as they are.

Think about it. The process of assigning a specific expert role to an LLM with billions of parameters, injecting complex library dependency information as context, and combining numerous constraints to control the generation of optimal code is clearly in a realm that ‘cannot practically be performed in the human mind.’ This is a crucial standard for recognizing patent eligibility in the guidelines and case law of the U.S. Patent and Trademark Office (USPTO).

 

🌍 A Comparative Look at Key Examination Standards of Major Patent Offices

The patentability of prompt technology is not assessed uniformly across all countries. If you are considering international filing, it is crucial to understand the subtle differences in perspective among major patent offices.

1. USPTO (United States Patent and Trademark Office) – Emphasis on the Abstract Idea Exception

The USPTO strictly applies the Alice/Mayo two-step test, which originated from Supreme Court case law. Instructions or general linguistic expressions that merely replace human thought processes can be dismissed as “abstract ideas.” However, if it can be demonstrated that the prompt is linked to a concrete technical implementation (e.g., improving model accuracy, optimizing specific hardware operations), there is a chance of it being recognized as patent-eligible subject matter.

2. EPO (European Patent Office) – Focus on Technical Effect

The EPO assesses based on “technical character” and “technical effect.” Simply presenting data input or linguistic rules is considered to lack inventive step, but if the prompt structure serves as a means to solve a technical problem (e.g., improving computational efficiency, optimizing memory usage, enhancing interaction with a specific device), it can be recognized as patent-eligible.

3. KIPO (Korean Intellectual Property Office) – Emphasis on Substantive Requirements for Software Inventions

KIPO places importance on the traditional requirement of “a creation of a technical idea utilizing the laws of nature.” Therefore, a prompt as a mere sentence or linguistic rule is not considered a technical idea, but if it is shown to be combined with a specific algorithm, hardware, or system to produce a concrete technical result, it can be recognized as an invention. In Korean practice, presenting a concrete system structure or processing flow is particularly persuasive.

Key Comparison Summary

Patent Office	Key Requirement
USPTO (U.S.)	Emphasis on ‘concrete technical implementation’ to avoid the abstract idea exception
EPO (Europe)	Proof of ‘technical effect’ is key; simple data manipulation is insufficient
KIPO (Korea)	Must be a technical idea using laws of nature + emphasis on systemic/structural implementation

⚠️ Implications for International Filing
The same “LLM prompt” technology could be at risk of being dismissed as an “abstract business method” in the United States, a “non-technical linguistic rule” in Europe, and a “mere idea” in Korea. Therefore, when considering international filing, a strategy that clearly articulates the ‘concrete system architecture’ and ‘measurable technical effects’ throughout the specification is essential as a common denominator.

 

🧮 A Practical Guide to Drafting Patent Claims (Detailed)

So, how should you draft patent claims to avoid the ‘human intervention’ attack and clearly establish that it is a ‘technical invention’? Let’s take a closer look at four key strategies.

1. Set the subject as the ‘computer (processor),’ not the ‘person.’

This is the most crucial step in shifting the focus of the invention from the ‘user’s mental activity’ to the ‘machine’s technical operation.’ It must be specified that all steps of the claim are performed by computer hardware (processor, memory, etc.).

• Bad 👎: A method where a user specifies a persona to an LLM and generates code.
• Good 👍: A step where a processor, upon receiving a user’s input, assigns a professional persona for a specific programming language to the LLM.

2. Specify the prompt as ‘structured data.’

Instead of abstract expressions like ‘natural language prompt,’ you need to clarify that it is a concrete data structure processed by the computer. This shows that the invention is not just a simple idea.

• Bad 👎: A step of providing a natural language prompt to the LLM.
• Good 👍: A step of generating and providing to the LLM a machine-readable context schema that includes library names and version constraints.

3. Claim ‘system performance improvement,’ not the result.

Instead of subjective results like ‘good code,’ you must specify objective and measurable effects that substantially improve the computer’s functionality. This is the core of ‘technical effect.’

• Bad 👎: A step of generating optimized code.
• Good 👍: A step of controlling the LLM’s token generation probability through the schema to generate optimized code that reduces code compatibility errors and saves GPU memory usage.

4. Clarify the ‘automation’ process.

It should be specified that all processes after the initial input (data structuring, LLM control, result generation, etc.) are performed Automatically by the system without further human judgment, demonstrating that it is a reproducible technical process.

 

📜 Reinforced Claim Example

By integrating all the strategies described above, you can construct a reinforced patent claim as follows.

[Claim] A computer-implemented method for generating optimized code, comprising:

1. (a) parsing, by a processor, a user’s natural language input to generate a persona identifier defining an expert role for a specific programming language;
2. (b) generating, by the processor, by referencing said input and an external code repository, structured context data including library names, version constraints, and hardware memory usage limits;
3. (c) generating, by the processor, a control prompt including said persona identifier and structured context data and transmitting it to an LLM, thereby automatically controlling the internal token generation process of the LLM;
4. (d) receiving, from said controlled LLM, optimized code that satisfies said constraints and has a compilation error rate below a predefined threshold and reduced GPU memory usage.

→ This example, instead of focusing on a simple result, greatly increases the chances of patent registration by clarifying system-level measurable technical effects such as ‘reduced compilation error rate’ and ‘reduced GPU memory usage.’

 

Frequently Asked Questions ❓

Q: Can a simple prompt like "write a poem about a cat" be patented?

A: No, that in itself is just an idea and would be difficult to patent. The subject of a patent would be a technical method or system that uses a prompt with a specific data structure (e.g., a schema defining poetic devices, rhyme schemes) to control an LLM to generate a poem, resulting in less computational resource usage or more accurate generation of a specific style of poetry.

Q: What are some specific ‘technical effects’ of prompt technology?

A: Typical examples include reduced compilation error rates in code generation, savings in computational resources like GPU and memory, shorter response generation times, and improved output accuracy for specific data formats (JSON, XML, etc.). The important thing is that these effects must be measurable and reproducible.

Q: Do I need to draft claims differently for each country when filing internationally?

A: Yes, while the core strategy is the same, it is advantageous to tailor the emphasis to the points that each patent office values. For example, in a U.S. (USPTO) specification, you would emphasize the ‘concrete improvement of computer functionality,’ in Europe (EPO), the ‘technical effect through solving a technical problem,’ and in Korea (KIPO), the ‘concreteness of the system configuration and processing flow.’

In conclusion, there is a clear path to protecting AI prompts with patents. However, it requires a strategic approach that goes beyond the idea of ‘what to ask’ and clearly demonstrates ‘how to technically control and improve a computer system.’ I hope this article provides a small clue to turning your innovative ideas into powerful intellectual property. If you have any more questions, feel free to ask in the comments~ 😊

※ This blog post is intended for general informational purposes only and does not constitute legal advice on any specific matter. For individual legal issues, please consult a qualified professional.

LLM-Powered Patent Search from A to Z: From Basic Prompts to Advanced Strategy

 

Still Stumped by Patent Searches with LLMs? This post breaks down how to use the latest AI Large Language Models (LLMs) to maximize the accuracy and efficiency of your patent searches, including specific model selection methods and advanced ‘deep research’ prompting techniques.

Hi there! Have you ever spent days, or even weeks, lost in a sea of patent documents, trying to find that one piece of information you need? I’ve definitely been there. The anxiety of wondering, ‘Is my idea truly novel?’ can keep you up at night. But thanks to the latest Large Language Models (LLMs), the whole paradigm of patent searching is changing. It’s even possible for an AI to conduct its own ‘deep research’ by diving into multiple sources. Today, I’m going to share some practical examples of ‘prompt engineering’ that I’ve learned firsthand to help you unlock 200% of your LLM’s potential!

Prompt Engineering Tricks to Boost Accuracy by 200%

Choosing the right AI model is important, but the success of your patent search ultimately depends on how you ask your questions. That’s where ‘prompt engineering’ comes in. It’s the key to making the AI accurately grasp your intent and deliver the best possible results. Let’s dive into some real-world examples.

Heads Up!
LLMs are not perfect. They can sometimes confidently present false information, a phenomenon known as ‘hallucination.’ It’s crucial to get into the habit of cross-referencing any patent numbers or critical details the AI provides with an official database.

 

1. Using Chain-of-Thought for Step-by-Step Reasoning

When you have a complex analysis task, asking the AI to ‘show its work’ by thinking step-by-step can reduce logical errors and improve accuracy.

Prompt Example:
Analyze the validity of a patent for an ‘autonomous driving technology that fuses camera and LiDAR sensor data’ by following these steps.

Step 1: Define the core technical components (camera, LiDAR, data fusion).
Step 2: Based on the defined components, generate 5 sets of search keywords for the USPTO database.
Step 3: From the search results, select the 3 most similar prior art patents.
Step 4: Compare the key claims of the selected patents with our technology, and provide your final opinion on the patentability of our tech.

 

2. Using Real-Time External Information (RAG & ReAct)

LLMs only know information up to their last training date. To get the latest patent data, you need to instruct them to search external databases in real-time.

Prompt Example:
You are a patent analyst. Using your search tool, find all patent publications on KIPRIS related to ‘Quantum Dot Displays’ published since January 1, 2024.

1. Organize the list of patents by application number, title of invention, and applicant.
2. Summarize the overall technology trends and analyze the core technical focus of the top 3 applicants.
3. Based on your analysis, predict which technologies in this field are likely to be promising over the next two years.

 

3. Activating the “Deep Research” Function

The latest LLMs can do more than just a single search. They have ‘deep research’ capabilities that can synthesize information from multiple websites, academic papers, and technical documents to create a comprehensive report, much like a human researcher.

Prompt Example:
Activate your deep research function. Write an in-depth report on the global R&D trends for ‘next-generation semiconductor materials using Graphene.’ The report must include the following:

1. The main challenges of the current technology and the latest research trends aimed at solving them (reference and summarize at least 3 reputable academic papers or tech articles).
2. An analysis of the top 5 companies and research institutions leading this field and their key patent portfolios.
3. The expected technology development roadmap and market outlook for the next 5 years.
4. Clearly cite the source (URL) for all information referenced in the report.

 

4. Exploring Multiple Paths (Tree of Thoughts)

This is useful for solving strategic problems with no single right answer, like designing around a patent or charting a new R&D direction. You have the AI explore and evaluate multiple possible scenarios.

Prompt Example:
Propose three new design concepts for a ‘secondary battery electrode structure’ that do not infringe on claim 1 of U.S. Patent ‘US 1234567 B2’.

1. For each design, clearly explain which elements of the original patent were changed and how.
2. Evaluate the technical advantages, expected performance, and potential drawbacks of each design.
3. Select the design you believe has the highest likelihood of avoiding infringement and achieving commercial success, and provide a detailed argument for your choice.

💡 Pro Tip!
The common thread in all great prompts is that they give the AI a clear ‘role,’ explain the ‘context,’ and demand a ‘specific output format.’ Just remembering these three things will dramatically improve your results.

💡

LLM Patent Search: Key Takeaways

Assign a Role: Give the AI a specific expert role, like “You are a patent attorney.”

Step-by-Step Thinking: For complex analyses, instruct the AI to use step-by-step reasoning (CoT) to improve logical accuracy.

Advanced Strategies:

Use Deep Research and Tree of Thoughts to generate expert-level reports.

Cross-Verification is a Must: Always be aware of AI hallucinations and verify important information against original sources.

One smart question can save you hours of work.

Frequently Asked Questions

Q: Is the ‘deep research’ function available on all LLMs?

A: No, not yet. It’s more of an advanced feature typically found in the latest premium versions of LLMs like Perplexity, Gemini, and ChatGPT. However, you can mimic a similar effect by using the standard search function and asking questions in multiple, sequential steps.

Q: Can I trust the search results from an LLM 100%?

A: No, you absolutely cannot. An LLM is a powerful assistant, not a substitute for a qualified expert’s final judgment. Due to hallucinations, it can invent patent numbers or misrepresent content. It is essential to always verify its findings against the original documents and have them reviewed by a professional.

Q: Prompt engineering seems complicated. Where should I start?

A: An easy way to start is by modifying the examples shown today. Just applying three techniques—’assigning a role,’ ‘specifying the format,’ and ‘requesting step-by-step thinking’—will dramatically improve the quality of your results.

Patent searching is no longer the tedious, uphill battle it once was. How you wield the powerful tool of LLMs can change the speed of your R&D and business. I hope you’ll use the tips I’ve shared today to create smarter innovations with AI. If you have any more questions, feel free to ask in the comments!

마틴 파울러가 말하는 LLM과 소프트웨어 개발의 미래: '환각'은 결함이 아니다? Martin Fowler on the Future of LLM and Software Development: Is 'Hallucination' Not a Flaw?

 

LLM의 '환각'이 결함이 아니라고?
Is LLM's 'Hallucination' Not a Flaw?
세계적인 소프트웨어 개발 사상가 마틴 파울러가 제시하는 LLM 시대의 개발 패러다임! 그의 날카로운 통찰을 통해 '비결정성'과 새로운 보안 위협 등 개발자가 마주할 미래를 미리 확인해 보세요.
The development paradigm for the LLM era presented by world-renowned software development thinker Martin Fowler! Get a preview of the future developers will face, including 'non-determinism' and new security threats, through his sharp insights.

안녕하세요! 요즘 너나 할 것 없이 AI, 특히 LLM(대규모 언어 모델)을 업무에 활용하고 있죠. 코드를 짜게 하거나, 아이디어를 얻거나, 심지어는 복잡한 개념을 설명해달라고 하기도 하고요. 저 역시 LLM의 편리함에 푹 빠져 지내고 있는데요, 문득 이런 생각이 들더라고요. '과연 우리는 이 도구를 제대로 이해하고 사용하고 있는 걸까?'
Hello! Nowadays, everyone is using AI, especially LLMs (Large Language Models), for work. We make them write code, get ideas, or even ask them to explain complex concepts. I'm also deeply immersed in the convenience of LLMs, but a thought suddenly struck me: 'Are we truly understanding and using this tool correctly?'

이런 고민의 와중에 소프트웨어 개발 분야의 세계적인 구루, 마틴 파울러(Martin Fowler)가 최근 LLM과 소프트웨어 개발에 대한 생각을 정리한 글을 읽게 되었습니다. 단순히 'LLM은 대단해!' 수준을 넘어, 그 본질적인 특성과 우리가 앞으로 마주하게 될 변화에 대한 깊이 있는 통찰이 담겨 있었죠. 오늘은 여러분과 함께 그의 생각을 따라가 보려고 합니다. 😊
While pondering this, I came across an article by Martin Fowler, a world-renowned guru in the software development field, who recently summarized his thoughts on LLMs and software development. It went beyond a simple 'LLMs are amazing!' level, offering deep insights into their fundamental nature and the changes we will face. Today, I'd like to explore his thoughts with you. 😊

LLM and Software Development

 

마틴 파울러, LLM의 현주소를 말하다 🤔
Martin Fowler on the Current State of LLMs 🤔

마틴 파울러는 먼저 현재 AI 산업이 명백한 '버블' 상태에 있다고 진단합니다. 하지만 역사적으로 모든 기술 혁신이 그래왔듯, 버블이 꺼진 후에도 아마존처럼 살아남아 새로운 시대를 여는 기업이 나타날 것이라고 봤어요. 중요한 건, 지금 단계에서는 프로그래밍의 미래나 특정 직업의 안정성에 대해 누구도 확실히 알 수 없다는 점입니다.
Martin Fowler first diagnoses the current AI industry as being in a clear 'bubble' state. However, as with all technological innovations historically, he believes that even after the bubble bursts, companies like Amazon will survive and usher in a new era. The important thing is that at this stage, no one can be certain about the future of programming or the job security of specific professions.

그래서 그는 섣부른 예측보다는 각자 LLM을 직접 사용해보고, 그 경험을 적극적으로 공유하는 실험적인 자세가 중요하다고 강조합니다. 우리 모두가 새로운 도구를 탐험하는 개척자가 되어야 한다는 의미겠죠?
Therefore, he emphasizes that an experimental attitude of personally using LLMs and actively sharing those experiences is more important than making hasty predictions. This implies that we all need to become pioneers exploring this new tool, right?

💡 알아두세요!
💡 Good to know!
파울러는 최근 LLM 활용에 대한 설문조사들이 실제 사용 흐름을 제대로 반영하지 못할 수 있다고 지적했어요. 다양한 모델의 기능 차이도 크기 때문에, 다른 사람의 의견보다는 자신의 직접적인 경험을 믿는 것이 더 중요해 보입니다.
Fowler pointed out that recent surveys on LLM usage may not accurately reflect actual usage patterns. Since there are also significant differences in the capabilities of various models, it seems more important to trust your own direct experience rather than the opinions of others.

 

LLM의 환각: 결함이 아닌 본질적 특징 🧠
LLM Hallucination: An Intrinsic Feature, Not a Flaw 🧠

이번 글에서 가장 흥미로웠던 부분입니다. 파울러는 LLM이 사실이 아닌 정보를 그럴듯하게 만들어내는 '환각(Hallucination)' 현상을 단순한 '결함'이 아니라 '본질적인 특성'으로 봐야 한다고 주장합니다. 정말 충격적이지 않나요? LLM은 결국 '유용성이 있는 환각을 생성하기 위한 도구'라는 관점입니다.
This was the most interesting part of the article for me. Fowler argues that the 'hallucination' phenomenon, where LLMs create plausible but untrue information, should be seen as an 'intrinsic feature' rather than a mere 'flaw'. Isn't that shocking? The perspective is that LLMs are ultimately 'tools for generating useful hallucinations'.

이런 관점에서 보면, 우리는 LLM의 답변을 맹목적으로 신뢰해서는 안 됩니다. 오히려 동일한 질문을 여러 번, 표현을 바꿔가며 던져보고 답변의 일관성을 확인하는 작업이 필수적입니다. 특히 숫자 계산과 같이 결정적인 답이 필요한 문제에 LLM을 직접 사용하려는 시도는 적절하지 않다고 덧붙였습니다.
From this viewpoint, we should not blindly trust the answers from LLMs. Instead, it is essential to ask the same question multiple times with different phrasing to check for consistency in the answers. He added that attempting to use LLMs directly for problems requiring definitive answers, such as numerical calculations, is not appropriate.

⚠️ 주의하세요!
⚠️ Be careful!
파울러는 LLM을 '주니어 개발자'에 비유하는 것에 강하게 비판합니다. LLM은 "모든 테스트 통과!"라고 자신 있게 말하면서 실제로는 테스트를 실패시키는 코드를 내놓는 경우가 흔하죠. 만약 인간 동료가 이런 행동을 반복한다면, 신뢰를 잃고 인사 문제로 이어질 수준의 심각한 결함이라는 것입니다. LLM은 동료가 아닌, 강력하지만 실수를 저지를 수 있는 '도구'로 인식해야 합니다.
Fowler strongly criticizes the analogy of an LLM to a 'junior developer'. LLMs often confidently state "All tests passed!" while providing code that actually fails tests. If a human colleague were to do this repeatedly, it would be a serious flaw leading to a loss of trust and personnel issues. LLMs should be recognized not as colleagues, but as powerful 'tools' that can make mistakes.

 

소프트웨어 공학, '비결정성' 시대로의 전환 🎲
Software Engineering's Shift to an Era of 'Non-Determinism' 🎲

전통적인 소프트웨어 공학은 '결정론적'인 세계 위에 세워져 있었습니다. '2+2'를 입력하면 '4'가 나와야 하듯, 모든 것은 예측 가능하고 일관적이어야 했죠. 예상과 다른 결과는 '버그'로 취급되어 즉시 수정되었습니다.
Traditional software engineering was built on a 'deterministic' world. Just as inputting '2+2' must yield '4', everything had to be predictable and consistent. Unexpected results were treated as 'bugs' and fixed immediately.

하지만 LLM의 등장은 이러한 패러다임을 근본적으로 바꾸고 있습니다. 파울러는 LLM이 소프트웨어 공학에 '비결정성(Non-Determinism)'을 도입하는 전환점이 될 것이라고 진단합니다. 동일한 요청에도 LLM은 미묘하게 다른 결과물을 내놓을 수 있으며, 그럴듯해 보이는 코드 안에 치명적인 오류를 숨겨놓기도 합니다.
However, the emergence of LLMs is fundamentally changing this paradigm. Fowler diagnoses that LLMs will be a turning point, introducing 'Non-Determinism' into software engineering. Even with the same request, an LLM can produce subtly different outputs and may hide critical errors within plausible-looking code.

이제 개발자의 역할은 단순히 코드를 작성하는 것을 넘어, LLM이 만들어낸 불확실한 결과물을 비판적으로 검증하고 관리하는 능력이 더욱 중요해졌습니다. 아래 표로 그 차이를 간단히 정리해봤습니다.
Now, the role of a developer has become more about the ability to critically verify and manage the uncertain outputs generated by LLMs, going beyond simply writing code. I've summarized the differences in the table below.

구분 Category	전통적 소프트웨어 (결정적) Traditional Software (Deterministic)	LLM 기반 소프트웨어 (비결정적) LLM-based Software (Non-deterministic)
결과 예측성 Result Predictability	동일 입력, 동일 결과 보장 Same input, same output guaranteed	동일 입력에도 다른 결과 가능 Different outputs possible for the same input
오류의 정의 Definition of Error	예측을 벗어난 모든 동작 (버그) Any behavior deviating from prediction (Bug)	결과의 불확실성 (본질적 특성) Uncertainty of results (Intrinsic feature)
개발자 역할 Developer's Role	정확한 로직 구현 및 디버깅 Implementing precise logic and debugging	결과물 검증 및 불확실성 관리 Verifying outputs and managing uncertainty

 

피할 수 없는 위협: 보안 문제 🔐
The Unavoidable Threat: Security Issues 🔐

마지막으로 파울러는 LLM이 소프트웨어 시스템의 공격 표면을 광범위하게 확대한다는 심각한 경고를 던집니다. 특히 브라우저 에이전트와 같이 비공개 데이터 접근, 외부 통신, 신뢰할 수 없는 콘텐츠 노출이라는 '치명적 삼중' 위험을 가진 도구들은 근본적으로 안전하게 만들기 어렵다는 것이 그의 의견입니다.
Finally, Fowler issues a serious warning that LLMs significantly expand the attack surface of software systems. He opines that tools with the 'lethal triple' risk of accessing private data, communicating externally, and being exposed to untrusted content, such as browser agents, are fundamentally difficult to secure.

예를 들어, 웹 페이지에 인간의 눈에는 보이지 않는 명령어를 숨겨 LLM을 속이고, 이를 통해 민감한 개인 정보를 유출하도록 유도하는 공격이 가능해집니다. 개발자들은 이제 코드의 기능뿐만 아니라, LLM과 상호작용하는 모든 과정에서 발생할 수 있는 새로운 보안 취약점을 고려해야 합니다.
For example, it becomes possible to trick an LLM by hiding commands invisible to the human eye on a web page, thereby inducing it to leak sensitive personal information. Developers must now consider not only the functionality of their code but also new security vulnerabilities that can arise in all processes interacting with LLMs.

💡

마틴 파울러의 LLM 핵심 인사이트
Martin Fowler's Core LLM Insights

환각은 본질:
Hallucination is Intrinsic: LLM의 환각은 '결함'이 아닌 '본질적 특징'으로 이해해야 합니다.
LLM's hallucination must be understood as an 'intrinsic feature,' not a 'flaw.'

비결정성의 시대:
The Era of Non-Determinism: 소프트웨어 공학이 예측 불가능성을 관리하는 시대로 진입했습니다.
Software engineering has entered an era of managing unpredictability.

검증은 필수:
Verification is a Must:

LLM의 결과물은 주니어 개발자가 아닌, 검증이 필수적인 '도구'의 산출물입니다.
The output of an LLM is not that of a junior developer, but the product of a 'tool' that requires mandatory verification.

보안 위협:
Security Threats: LLM은 시스템의 공격 표면을 넓히는 새로운 보안 변수입니다.
LLMs are a new security variable that broadens a system's attack surface.

LLM을 현명하게 활용하려면 비판적 사고와 지속적인 실험이 중요합니다.
To use LLMs wisely, critical thinking and continuous experimentation are crucial.

자주 묻는 질문 ❓
Frequently Asked Questions ❓

Q: 마틴 파울러가 '환각'을 결함이 아닌 본질로 봐야 한다고 말하는 이유는 무엇인가요?
Q: Why does Martin Fowler say that 'hallucination' should be seen as an intrinsic feature, not a flaw?

A: LLM은 방대한 데이터를 기반으로 가장 그럴듯한 다음 단어를 예측하여 문장을 생성하는 모델이기 때문입니다. 이 과정에서 사실관계와 무관하게 매끄러운 문장을 만들어내는 '환각'은 자연스러운 결과물이며, 이 특성을 이해해야 LLM을 올바르게 활용할 수 있다는 의미입니다.
A: This is because LLMs are models that generate sentences by predicting the most plausible next word based on vast amounts of data. In this process, 'hallucination,' which creates fluent sentences regardless of factual accuracy, is a natural outcome. Understanding this characteristic is key to using LLMs correctly.

Q: 소프트웨어 공학의 '비결정성'이란 무엇을 의미하며, 왜 중요한가요?
Q: What does 'non-determinism' in software engineering mean, and why is it important?

A: '비결정성'이란 동일한 입력에 대해 항상 동일한 결과가 나오지 않는 특성을 의미합니다. 전통적인 소프트웨어는 100% 예측 가능해야 했지만, LLM은 같은 질문에도 다른 답변을 줄 수 있습니다. 이 불확실성을 이해하고 관리하는 것이 LLM 시대 개발자의 핵심 역량이 되었습니다.
A: 'Non-determinism' refers to the characteristic where the same input does not always produce the same output. While traditional software had to be 100% predictable, an LLM can give different answers to the same question. Understanding and managing this uncertainty has become a core competency for developers in the age of LLMs.

Q: LLM이 생성한 코드를 신뢰하고 바로 사용해도 될까요?
Q: Can I trust and use the code generated by an LLM immediately?

A: 아니요, 절대 안 됩니다. 마틴 파울러는 LLM이 그럴듯하지만 작동하지 않거나, 보안에 취약한 코드를 생성할 수 있다고 경고합니다. 생성된 코드는 반드시 개발자가 직접 검토, 테스트, 검증하는 과정을 거쳐야 합니다.
A: No, absolutely not. Martin Fowler warns that LLMs can generate code that looks plausible but doesn't work or is insecure. The generated code must be reviewed, tested, and verified by a developer.

Q: LLM을 사용하면 왜 보안 위협이 커지나요?
Q: Why do security threats increase with the use of LLMs?

A: LLM은 외부 데이터와 상호작용하고, 때로는 민감한 정보에 접근할 수 있기 때문입니다. 악의적인 사용자가 웹사이트나 입력값에 보이지 않는 명령어를 숨겨 LLM을 조종(프롬프트 인젝션)하여 정보를 유출하거나 시스템을 공격하는 새로운 형태의 보안 위협이 발생할 수 있습니다.
A: Because LLMs interact with external data and can sometimes access sensitive information. Malicious users can hide invisible commands in websites or inputs to manipulate the LLM (prompt injection), leading to new types of security threats such as data leakage or system attacks.

마틴 파울러의 통찰은 LLM이라는 새로운 도구를 어떻게 바라보고 사용해야 하는지에 대한 중요한 가이드를 제시합니다. 단순히 편리한 코드 생성기를 넘어, 우리 개발 환경의 근본적인 패러다임을 바꾸는 존재임을 인식해야 합니다. 그의 조언처럼, 두려워하거나 맹신하기보다는 적극적으로 실험하고 경험을 공유하며 이 거대한 변화의 물결에 현명하게 올라타야 할 때입니다.
Martin Fowler's insights provide an important guide on how to view and use the new tool that is the LLM. We must recognize it not just as a convenient code generator, but as an entity that is changing the fundamental paradigm of our development environment. As he advises, now is the time to wisely ride this massive wave of change by experimenting and sharing experiences, rather than fearing or blindly trusting it.

여러분은 LLM에 대해 어떻게 생각하시나요? 개발 과정에서 겪었던 흥미로운 경험이 있다면 댓글로 공유해주세요! 😊
What are your thoughts on LLMs? If you have any interesting experiences from your development process, please share them in the comments! 😊

나는 지시한다, 고로 창작한다 - AI와 창작자의 새로운 관계, "I Direct, Therefore I Create" - The New Relationship Between AI and the Creator

AI에게 '지시'만 내린 사람, 과연 창작자일까요?
If You Only 'Direct' an AI, Are You Still the Creator?

최근 인공지능(AI)을 활용해 15초 분량의 짧은 영상을 만들어 보았습니다. 제가 한 것이라고는 오직 두 가지 프롬프트를 지시한 것뿐이었습니다.
I recently used artificial intelligence (AI) to create a short, 15-second video. All I did was provide two prompts.

👤 인물 캐릭터 생성 프롬프트
👤 Character Creation Prompt

“감정적으로 빈티지 마이크에 대고 노래하는 젊은 여성의 클로즈업 초상화. 은은하게 반짝이는 연한 하늘색 드레스를 입고 있으며, 메이크업은 부드럽고 감정 어린 눈빛과 붉은 입술을 강조한다. 머리는 꽃 장식이 달린 단정한 업스타일로 묶여 있다. 배경은 은은하게 빛나는 파란 커튼과 흐릿한 따뜻한 전구 조명이 아련하고 친밀한 무대 분위기를 만든다. 스타일은 사실적이고 시네마틱하며, 얼굴과 마이크에 초점을 맞춘 고해상도 디테일로 표현한다.”

"A close-up portrait of a young woman singing emotionally into a vintage microphone. She wears a sparkling light-blue dress with thin straps, and her makeup highlights her soft, expressive eyes and red lips. Her hair is styled in a loose elegant updo with a flower accessory. The background has softly glowing blue curtains and blurred warm string lights, creating a dreamy and intimate stage mood. The style should be photorealistic, cinematic, and highly detailed, focusing on her face and microphone."

🎵 가사와 음원 생성 프롬프트
🎵 Lyrics and Music Generation Prompt

“부드러운 피아노 선율에 맞추어 부르는 진심 어린 발라드. 목소리는 섬세하고 감정적이며, 첫사랑의 달콤쌉싸름한 기억을 회상한다. 가사는 한국어로 향수와 연약함, 그리고 그리움으로 가득 차 있다. 전체적인 분위기는 서정적이고 시네마틱하며, 잔잔한 리듬과 풍부한 표현력이 잃어버린 사랑의 아픔과 아름다움을 동시에 드러낸다.”

"A heartfelt ballad with soft piano melodies, sung in a tender and emotional voice. The song recalls the bittersweet memory of a first love — nostalgic, delicate, and filled with longing. The lyrics are written in Korean, filled with shades of nostalgia, fragility, and yearning. The mood is sentimental and cinematic, with a gentle rhythm and expressive dynamics that reveal both the pain and the beauty of lost love."

이 두 가지 지시에 따라 AI가 영상을 창작했고, 무료 버전이라 15초 길이로 제작되었습니다.
Following these two instructions, the AI created the video. Since I used a free version, it was produced as a 15-second clip.

예술계에서는 작품을 직접 제작하지 않더라도 창작 과정을 기획하거나 방향을 제시한 사람 또한 창작자로 인정된다고 합니다. 그렇다면 저 역시 창작자로 불릴 수 있을까요? 기술이 너무 빠르게 진화하는 오늘날, 이러한 물음을 곱씹을 틈도 없이 새로운 현실이 우리 앞에 펼쳐지고 있음을 실감합니다.
In the art world, it's said that even if someone doesn't physically create the work, the person who plans or directs the creative process is also recognized as a creator. If that's the case, can I too be called a creator? With technology evolving so rapidly today, I realize that a new reality is unfolding before us, leaving little time to ponder such questions.

예술의 오랜 질문: "창작자"는 누구인가?
Art's Enduring Question: Who is the "Creator"?

예술계에서는 오래전부터 “작품의 창작자란 누구인가”라는 논쟁이 이어져 왔습니다. 실질적으로 붓을 들지 않았더라도 작품의 기획, 구도, 개념을 제시한 사람은 ‘창작자’로 간주되어 왔습니다. 대표적으로 뒤샹(Marcel Duchamp)의 레디메이드(Ready-made) 예술이 보여주듯, 물건 자체를 만든 사람이 아닌, 그것을 예술의 맥락으로 끌어올린 사람이 창작자로 불렸습니다.
For a long time, the art world has debated the question, "Who is the creator of a work of art?" Even those who didn't physically hold the brush have been considered 'creators' if they provided the concept, composition, and plan for the piece. A prime example is Marcel Duchamp's "Ready-made" art. The person who elevated an object into the context of art, not the person who manufactured the object itself, was called the creator.

이는 창작의 본질이 행위의 물리적 구현이 아니라 ‘의도의 지시와 기획’에 있다는 철학적 전제를 반영합니다.
This reflects the philosophical premise that the essence of creation lies not in the physical act of making, but in the 'intention, direction, and planning.'

지시자 vs 창작자: 법과 현실의 간극
The Director vs. The Creator: A Gap Between Law and Reality

결국, 저는 AI에게 두 개의 프롬프트를 제시함으로써 창작 행위를 ‘지시·감독’했습니다. 예술철학적으로는 이러한 행위 자체가 창작의 본질에 가깝다고 볼 수 있습니다. 하지만 법적으로는, 아직까지 저와 같은 “지시자”의 지위가 창작자로 전면적으로 인정되지 않습니다. 다만 이 간극이야말로 법철학적 사유의 출발점이자, 기술 발전이 제기하는 새로운 질문입니다.
Ultimately, by providing two prompts to the AI, I 'directed and supervised' a creative act. From a philosophical standpoint, this act itself can be seen as close to the essence of creation. Legally, however, the status of a "director" like myself is not yet fully recognized as that of a creator. This very gap is the starting point for legal and philosophical inquiry and a new question posed by technological advancement.

따라서 저는 스스로에게 묻습니다.
Therefore, I ask myself:

“작품의 완성된 물리적 형태를 만든 것이 중요한가, 아니면 그 과정을 기획하고 방향을 정한 것이 중요한가?”
"Is it more important to have created the final physical form of a work, or to have planned and directed the process?"

기술은 이미 이 질문을 우리에게 강요하고 있으며, 법과 철학은 이제 그 답을 새롭게 모색해야 하는 시점에 와 있습니다.
Technology is already forcing this question upon us, and now law and philosophy must find a new answer.

아래에서 AI가 생성한 영상을 직접 확인해 보세요.
You can watch the video generated by the AI below.