ããªãã®Pythonã¢ããªã±ãŒã·ã§ã³ã®ã¡ã¢ãªåé¡ã解決ãã匷åãªå©ã£äºº
Pythonã¯èªåã¡ã¢ãªç®¡çæ©èœïŒã¬ããŒãžã³ã¬ã¯ã·ã§ã³ïŒãåããŠããŸãããããã§ãã¡ã¢ãªãªãŒã¯ãéå¹çãªã¡ã¢ãªäœ¿çšãšãã£ãåé¡ãçºçããããšããããŸããç¹ã«é·æéåäœãããµãŒããŒã¢ããªã±ãŒã·ã§ã³ãã倧éã®ããŒã¿ãåŠçãããããããã°ã©ã ã§ã¯ãã¡ã¢ãªã®åé¡ãããã©ãŒãã³ã¹ã®ããã«ããã¯ã«ãªã£ãããææªã®å Žåã¯ããã»ã¹ãã¯ã©ãã·ã¥ããåå ã«ãªã£ããããŸããð±
ãã®ãããªã¡ã¢ãªã®åé¡ãç¹å®ãã解決ããããã«Pythonãæšæºã§æäŸããŠãã匷åãªããŒã«ã tracemalloc
ã¢ãžã¥ãŒã«ã§ãããã®ã¢ãžã¥ãŒã«ã¯Python 3.4ã§å°å
¥ããã以éã®ããŒãžã§ã³ã§æ©èœæ¹åãç¶ããããŠããŸãã
ãã®ããã°èšäºã§ã¯ãtracemalloc
ã®åºæ¬çãªäœ¿ãæ¹ãããã¡ã¢ãªãªãŒã¯ã®ç¹å®ã詳现ãªã¡ã¢ãªåææ¹æ³ãŸã§ãå
·äœçãªã³ãŒãäŸã亀ããªãã詳ãã解説ããŠãããŸããðª
1. tracemallocãšã¯ïŒ ãªãã¡ã¢ãªãããã¡ã€ãªã³ã°ãéèŠãªã®ãïŒ
tracemalloc
ã¯ãPythonããã°ã©ã ãã¡ã¢ãªãã©ãã§ãã©ãã ã確ä¿ãããã远跡ïŒãã¬ãŒã¹ïŒããããã®ã¢ãžã¥ãŒã«ã§ããå
·äœçã«ã¯ã以äžã®æ
å ±ãæäŸããŠãããŸãã
- ãªããžã§ã¯ããå²ãåœãŠãããã³ãŒãäžã®å ŽæïŒãã¡ã€ã«åãšè¡çªå·ïŒã瀺ããã¬ãŒã¹ããã¯
- ãã¡ã€ã«åãè¡çªå·ããšã«éèšãããã¡ã¢ãªå²ãåœãŠçµ±èšïŒåèšãµã€ãºããããã¯æ°ãå¹³åãµã€ãºãªã©ïŒ
- ã¡ã¢ãªäœ¿çšéã®ã¹ãããã·ã§ããïŒç¹å®ã®æç¹ã§ã®ã¡ã¢ãªå²ãåœãŠç¶æ³ïŒãååŸããæ¯èŒããæ©èœ
ã¡ã¢ãªãããã¡ã€ãªã³ã°ãéèŠãªçç±ã¯ããã€ããããŸãã
- ã¡ã¢ãªãªãŒã¯ã®æ€åºãšä¿®æ£: ããã°ã©ã ãäžèŠã«ãªã£ãã¡ã¢ãªã解æŸãæããŠããç®æãç¹å®ããä¿®æ£ããããšã§ãã¢ããªã±ãŒã·ã§ã³ã®å®å®æ§ãåäžãããŸããã¡ã¢ãªãªãŒã¯ã¯ãåç §ãæå³ããæ®ãç¶ãã埪ç°åç §ãã解æŸãããã¹ããªãœãŒã¹ïŒãã¡ã€ã«ãã³ãã«ããããã¯ãŒã¯æ¥ç¶ãªã©ïŒãéããããŠããªãå Žåã«çºçããããã§ãã
- ããã©ãŒãã³ã¹æé©å: ã¡ã¢ãªãéå°ã«æ¶è²»ããŠããç®æãèŠã€ãåºããããã¡ã¢ãªå¹çã®è¯ãããŒã¿æ§é ãã¢ã«ãŽãªãºã ã«æ¹åããããšã§ãããã°ã©ã å šäœã®ããã©ãŒãã³ã¹ãåäžãããããšãã§ããŸãã
- ãªãœãŒã¹äœ¿çšéã®ææ¡: ã¢ããªã±ãŒã·ã§ã³ãã©ã®çšåºŠã®ã¡ã¢ãªãå¿ èŠãšããããæ£ç¢ºã«ææ¡ããããšã¯ãé©åãªã€ã³ãã©ã¹ãã©ã¯ãã£ã®éžå®ãããªãœãŒã¹å¶éã®ããç°å¢ïŒã³ã³ãããªã©ïŒã§ã®éçšã«ãããŠéèŠã§ãã
tracemalloc
ã䜿ãããšã§ããããã®èª²é¡ã«å¯ŸããŠå
·äœçãªããŒã¿ã«åºã¥ããã¢ãããŒããå¯èœã«ãªããŸããðµïžââïž
tracemalloc
ã¯Pythonã€ã³ã¿ãŒããªã¿ã«ãã£ãŠç¢ºä¿ãããã¡ã¢ãªã远跡ããŸããCæ¡åŒµã©ã€ãã©ãªïŒNumPyãPandasã®äžéšãªã©ïŒãå
éšã§çŽæ¥ç¢ºä¿ããã¡ã¢ãªã«ã€ããŠã¯è¿œè·¡ã§ããªãå ŽåããããŸãã
2. tracemallocã®åºæ¬çãªäœ¿ãæ¹
tracemalloc
ã䜿ãå§ããã®ã¯éåžžã«ç°¡åã§ããåºæ¬çãªæµãã¯ä»¥äžã®éãã§ãã
tracemalloc.start()
ãåŒã³åºããŠã¡ã¢ãªå²ãåœãŠã®ãã¬ãŒã¹ãéå§ããŸãã- ã¡ã¢ãªäœ¿çšéã調æ»ãããåŠçãå®è¡ããŸãã
tracemalloc.get_traced_memory()
ã§çŸåšã®ã¡ã¢ãªäœ¿çšéãšããŒã¯æã®ã¡ã¢ãªäœ¿çšéãååŸããŸããtracemalloc.stop()
ãåŒã³åºããŠãã¬ãŒã¹ãåæ¢ããŸãã
ç°¡åãªäŸãèŠãŠã¿ãŸãããã
import tracemalloc
import time
# ã¡ã¢ãªå²ãåœãŠã®ãã¬ãŒã¹ãéå§
tracemalloc.start()
# ã¡ã¢ãªãæ¶è²»ããå¯èœæ§ã®ããåŠç
my_list = [i for i in range(100000)]
my_dict = {str(i): i for i in range(10000)}
time.sleep(1) # äœããã®åŠçæéãã·ãã¥ã¬ãŒã
# çŸåšã®ã¡ã¢ãªäœ¿çšéãšããŒã¯æã®ã¡ã¢ãªäœ¿çšéãååŸ
current, peak = tracemalloc.get_traced_memory()
print(f"çŸåšã®ã¡ã¢ãªäœ¿çšé: {current / 1024:.2f} KiB")
print(f"ããŒã¯æã®ã¡ã¢ãªäœ¿çšé: {peak / 1024:.2f} KiB")
# ããŒã¯å€ããªã»ããããŠãç¹å®ã®åºéã®ããŒã¯ãèšæž¬ããããšãå¯èœ
tracemalloc.reset_peak()
another_list = ['x'] * 50000
current_after_reset, peak_after_reset = tracemalloc.get_traced_memory()
print(f"ãªã»ããåŸã®çŸåšã®ã¡ã¢ãªäœ¿çšé: {current_after_reset / 1024:.2f} KiB")
print(f"ãªã»ããåŸã®ããŒã¯æã®ã¡ã¢ãªäœ¿çšé: {peak_after_reset / 1024:.2f} KiB")
# ãã¬ãŒã¹ãåæ¢
tracemalloc.stop()
# åæ¢åŸã«åŒã³åºããšãšã©ãŒã«ãªã
try:
tracemalloc.get_traced_memory()
except RuntimeError as e:
print(f"\nãã¬ãŒã¹åæ¢åŸã®åŒã³åºããšã©ãŒ: {e}")
start()
é¢æ°ã«ã¯åŒæ° nframe
ãæå®ã§ããŸããããã¯ãåã¡ã¢ãªã¢ãã±ãŒã·ã§ã³ã«å¯ŸããŠä¿åããã¹ã¿ãã¯ãã¬ãŒã ã®æ°ãæå®ããŸããããã©ã«ã㯠1
ã§ãå²ãåœãŠãè¡ãããçŽè¿ã®ãã¬ãŒã ã®ã¿ãèšé²ããŸãããã詳现ãªãã¬ãŒã¹ããã¯ãå¿
èŠãªå Žåã¯ããã®å€ãå¢ãããŸãïŒäŸ: tracemalloc.start(25)
ïŒããã ãããã¬ãŒã æ°ãå¢ãããšãã¡ã¢ãªäœ¿çšéãšCPUãªãŒããŒããããå¢å ããŸããð
3. ã¹ãããã·ã§ããã䜿ã£ãã¡ã¢ãªãªãŒã¯ã®ç¹å® ðž
tracemalloc
ã®æã匷åãªæ©èœã®äžã€ããã¡ã¢ãªå²ãåœãŠã®ã¹ãããã·ã§ãããååŸããæ¯èŒããæ©èœã§ããããã«ãããç¹å®ã®åŠçåºéã§ã©ã®ãããªã¡ã¢ãªå²ãåœãŠãå¢å ããããã€ãŸãæœåšçãªã¡ã¢ãªãªãŒã¯ãç¹å®ããã®ã«åœ¹ç«ã¡ãŸãã
åºæ¬çãªæé ã¯ä»¥äžã®éãã§ãã
tracemalloc.start()
ã§ãã¬ãŒã¹ãéå§ããŸãã- 調æ»ãããåŠçã®åã«
tracemalloc.take_snapshot()
ãåŒã³åºããŠãæåã®ã¹ãããã·ã§ãããååŸããŸãã - ã¡ã¢ãªãªãŒã¯ãçãããåŠçãå®è¡ããŸãã
- åŠçã®åŸã«å床
tracemalloc.take_snapshot()
ãåŒã³åºããŠã2çªç®ã®ã¹ãããã·ã§ãããååŸããŸãã - 2çªç®ã®ã¹ãããã·ã§ããã®
compare_to()
ã¡ãœããã䜿ã£ãŠãæåã®ã¹ãããã·ã§ãããšã®å·®åãèšç®ããŸãã - å·®åæ å ±ãåæããŠãã¡ã¢ãªãå¢å ããŠããç®æãç¹å®ããŸãã
ã¡ã¢ãªãªãŒã¯ãã·ãã¥ã¬ãŒãããç°¡åãªäŸãèŠãŠã¿ãŸãããã
import tracemalloc
import os
# ã°ããŒãã«ãªã¹ãïŒã¡ã¢ãªãªãŒã¯ã®åå ïŒ
leaky_list = []
def memory_leaking_function():
"""æå³çã«ã¡ã¢ãªãªãŒã¯ãçºçãããé¢æ°"""
leaky_list.extend([os.urandom(1024) for _ in range(100)]) # 100KBãã€è¿œå
tracemalloc.start(10) # ãã¬ãŒã¹ããã¯çšã«ãã¬ãŒã æ°ãå¢ãã
print("æåã®ã¹ãããã·ã§ãããååŸ...")
snapshot1 = tracemalloc.take_snapshot()
# ã¡ã¢ãªãªãŒã¯ãçãããåŠçãå®è¡
for _ in range(5):
memory_leaking_function()
print("2çªç®ã®ã¹ãããã·ã§ãããååŸ...")
snapshot2 = tracemalloc.take_snapshot()
print("ã¹ãããã·ã§ããéã®å·®åãèšç®ã»è¡šç€º...")
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print("\n[ã¡ã¢ãªäœ¿çšéãå¢å ããäžäœ10ç®æ]")
for stat in top_stats[:10]:
print(stat)
print("\n[å·®åã®è©³çŽ°ãªãã¬ãŒã¹ããã¯]")
for stat in top_stats[:3]: # äžäœ3ã€ã®è©³çŽ°ã衚瀺
print(f"--- {stat} ---")
for line in stat.traceback.format():
print(line)
print("-" * 40)
tracemalloc.stop()
ãã®ã³ãŒããå®è¡ãããšãcompare_to()
ã®çµæãšã㊠StatisticDiff
ãªããžã§ã¯ãã®ãªã¹ããåŸãããŸããããã«ã¯ãåã³ãŒãè¡ã«ãããã¡ã¢ãªãããã¯ã®æ°ãšãµã€ãºã®å·®åãå«ãŸããŠããŸããåºåçµæãããmemory_leaking_function
å
ã®ãªã¹ããžã®èŠçŽ è¿œå ïŒleaky_list.extend(...)
ã®è¡ïŒã§ã¡ã¢ãªäœ¿çšéã倧å¹
ã«å¢å ããŠããããšã確èªã§ããã¯ãã§ãã
compare_to()
ã statistics()
ã¡ãœããã®ç¬¬äžåŒæ° (key_type
) ã«ã¯ãéèšã»ãœãŒãã®ããŒãæå®ããŸãããã䜿ãããããŒã¯ä»¥äžã®éãã§ãã
'lineno'
: ãã¡ã€ã«åãšè¡çªå·ã§éèšãã¡ã¢ãªãæ¶è²»ããŠããå ·äœçãªã³ãŒãè¡ãç¹å®ããããã'filename'
: ãã¡ã€ã«åã§éèšãã©ã®ã¢ãžã¥ãŒã«ãã¡ã¢ãªãå€ã䜿ã£ãŠããããææ¡ããã®ã«åœ¹ç«ã€ã'traceback'
: å®å šãªãã¬ãŒã¹ããã¯ã§éèšãåãã³ãŒã«ã¹ã¿ãã¯ããã®å²ãåœãŠãã°ã«ãŒãåã§ããããåºåãå€ããªããã¡ã
ã¡ã¢ãªãªãŒã¯èª¿æ»ã§ã¯ããŸã 'lineno'
ã§ããããã€ããå¿
èŠã«å¿ã㊠'traceback'
ã§è©³çŽ°ãªåŒã³åºãçµè·¯ã確èªããã®ãå¹æçã§ããð¡
4. ã¹ãããã·ã§ããã®é«åºŠãªåæãšãã£ã«ã¿ãªã³ã° ð¬
ååŸããã¹ãããã·ã§ããã¯ãåã«æ¯èŒããã ãã§ãªãããã詳现ãªåæãè¡ãããšãã§ããŸããSnapshot
ãªããžã§ã¯ãã«ã¯ statistics()
ã¡ãœããããããããã䜿ããšãã®ã¹ãããã·ã§ããæç¹ã§ã®ã¡ã¢ãªå²ãåœãŠçµ±èšãååŸã§ããŸãã
import tracemalloc
import numpy as np # äŸãšããŠNumPyã䜿çš
tracemalloc.start(5) # å°ãå€ãã«ãã¬ãŒã ãä¿å
# äœããã®åŠç
data = np.random.rand(1000, 1000)
intermediate_list = [str(i) * 10 for i in range(5000)]
result = {"data": data, "info": intermediate_list[:100]}
snapshot = tracemalloc.take_snapshot()
# ãã¡ã€ã«åãšè¡çªå·ã§çµ±èšæ
å ±ãååŸããäžäœ5件ã衚瀺
print("[ãã¡ã€ã«åãšè¡çªå·å¥ããã5]")
top_lineno = snapshot.statistics('lineno')
for stat in top_lineno[:5]:
print(stat)
# ãã¡ã€ã«åã§çµ±èšæ
å ±ãååŸããäžäœ5件ã衚瀺
print("\n[ãã¡ã€ã«åå¥ããã5]")
top_filename = snapshot.statistics('filename')
for stat in top_filename[:5]:
print(stat)
# ãã¬ãŒã¹ããã¯ã§çµ±èšæ
å ±ãååŸããäžäœ3件ã®è©³çŽ°ã衚瀺
print("\n[ãã¬ãŒã¹ããã¯å¥ããã3 (詳现)]")
top_traceback = snapshot.statistics('traceback')
for stat in top_traceback[:3]:
print(f"--- {stat} ---")
for line in stat.traceback.format():
print(line)
print("-" * 40)
tracemalloc.stop()
statistics()
ã¡ãœãã㯠Statistic
ãªããžã§ã¯ãã®ãªã¹ããè¿ããŸããå Statistic
ãªããžã§ã¯ãã¯ä»¥äžã®å±æ§ãæã£ãŠããŸãã
traceback
: ã¡ã¢ãªå²ãåœãŠå ã®ãã¬ãŒã¹ãã㯠(Traceback
ã€ã³ã¹ã¿ã³ã¹)ãsize
: ãã®ãã¬ãŒã¹ããã¯ããå²ãåœãŠãããã¡ã¢ãªã®åèšãµã€ãºïŒãã€ãåäœïŒãcount
: ãã®ãã¬ãŒã¹ããã¯ããå²ãåœãŠãããã¡ã¢ãªãããã¯ã®æ°ã
ãã£ã«ã¿ãªã³ã°ã«ãããã€ãºé€å»
tracemalloc
èªäœã®å
éšåäœãããããã°ããŒã«ãç¹å®ã®ã©ã€ãã©ãªã«ããã¡ã¢ãªå²ãåœãŠããã¬ãŒã¹çµæã«å«ãŸããããšããããŸãããããã¯ã¡ã¢ãªãªãŒã¯èª¿æ»ã®éã«ã¯ãã€ãºãšãªãå Žåãããããããã£ã«ã¿ãªã³ã°ããŠé€å€ãããå ŽåããããŸãã
ã¹ãããã·ã§ããã® filter_traces()
ã¡ãœãããš Filter
ã¯ã©ã¹ïŒãŸãã¯Python 3.6以éã§ã¯ DomainFilter
ãïŒã䜿ãããšã§ããã¬ãŒã¹æ
å ±ããã£ã«ã¿ãªã³ã°ã§ããŸãã
Filter
ã¯ã©ã¹ã®äž»ãªåŒæ°ã¯ä»¥äžã®éãã§ãã
inclusive
:True
ãªãæå®ãããã¿ãŒã³ã«äžèŽãããã®ãå«ããïŒã€ã³ã¯ã«ãŒã·ãïŒãFalse
ãªãäžèŽãããã®ãé€å€ããïŒãšã¯ã¹ã¯ã«ãŒã·ãïŒãfilename_pattern
: ãã¡ã€ã«åã®ãã¿ãŒã³ïŒfnmatch
圢åŒã®ã¯ã€ã«ãã«ãŒã䜿çšå¯èœïŒãlineno
: ç¹å®ã®è¡çªå·ïŒãªãã·ã§ã³ïŒãall_frames
:True
ãªããã¬ãŒã¹ããã¯å ã®å šãã¬ãŒã ã察象ã«ãFalse
(ããã©ã«ã) ãªãæãæ°ãããã¬ãŒã ã®ã¿ã察象ã«ãã£ã«ã¿ãªã³ã°ãdomain
: (Python 3.6+) ã¡ã¢ãªã¢ãã¬ã¹ç©ºéãã¡ã€ã³ãéåžžã¯Pythonå éšã¡ã¢ãªã瀺ã0
ã䜿çšã
import tracemalloc
import os
import fnmatch # ã¯ã€ã«ãã«ãŒããããã³ã°çš
tracemalloc.start(10)
# ãããštracemallocèªèº«ã«é¢ããæäœãå
¥ããŠã¿ã
current, peak = tracemalloc.get_traced_memory()
print(f"çŸåšã® tracemalloc å
éšã¡ã¢ãª: {tracemalloc.get_tracemalloc_memory()} bytes")
# äœããã®åŠç
my_data = [os.urandom(512) for _ in range(200)]
import collections # collections ã¢ãžã¥ãŒã«ã䜿ã£ãŠã¿ã
my_deque = collections.deque(range(1000))
snapshot = tracemalloc.take_snapshot()
print(f"\nãã£ã«ã¿åã®ãã¬ãŒã¹æ°: {len(snapshot.traces)}")
# --- ãã£ã«ã¿ãªã³ã°ã®äŸ ---
# 1. tracemallocèªèº«ã®ã³ãŒããé€å€ãããã£ã«ã¿
filter1 = tracemalloc.Filter(inclusive=False, filename_pattern=tracemalloc.__file__)
# 2. ç¹å®ã®ãã£ã¬ã¯ããªé
äžã®ãã¡ã€ã«ã®ã¿ãå«ãããã£ã«ã¿ (äŸ: site-packagesãé€å€)
# 泚æ: ãã¹åºåãæåã¯ç°å¢äŸåã®å¯èœæ§ããããã os.sep ã䜿ãã®ãå
ç¢
import sys
site_packages_path = None
for p in sys.path:
if 'site-packages' in p:
site_packages_path = p
break
filters = [filter1]
if site_packages_path:
# site-packages å
ã®ãã¡ã€ã«ãé€å€ãããã£ã«ã¿ãè¿œå
# fnmatch ã§ãã£ã¬ã¯ããªãæ±ãå Žåã¯æ«å°Ÿã« * ãã€ããããšãå€ã
filter_exclude_site_packages = tracemalloc.Filter(inclusive=False, filename_pattern=os.path.join(site_packages_path, "*"))
filters.append(filter_exclude_site_packages)
print(f"é€å€ãã£ã«ã¿ (site-packages): {filter_exclude_site_packages.filename_pattern}")
# 3. ç¹å®ã®ãã¡ã€ã«ãã¿ãŒã³ã®ã¿å«ãã (äŸ: èªåã®ãããžã§ã¯ãã³ãŒã 'my_project/' é
äžãªã©)
# ããã§ã¯äŸãšããŠçŸåšã®ã¹ã¯ãªãããã¡ã€ã«ã®ã¿ãå«ãããã£ã«ã¿ãäœæ
current_script_file = __file__
filter_only_this_script = tracemalloc.Filter(inclusive=True, filename_pattern=current_script_file)
# filters.append(filter_only_this_script) # å¿
èŠã«å¿ããŠæå¹å
# ãã£ã«ã¿ãé©çš
filtered_snapshot = snapshot.filter_traces(filters)
print(f"ãã£ã«ã¿åŸã®ãã¬ãŒã¹æ°: {len(filtered_snapshot.traces)}")
# ãã£ã«ã¿åŸã®çµ±èšæ
å ±ã衚瀺
print("\n[ãã£ã«ã¿åŸã®ã¡ã¢ãªäœ¿çšéããã5 (lineno)]")
top_stats_filtered = filtered_snapshot.statistics('lineno')
for stat in top_stats_filtered[:5]:
print(stat)
tracemalloc.stop()
è€æ°ã®ãã£ã«ã¿ããªã¹ãã§æž¡ããšããŸãå
šãŠã® inclusive=True
ã®ãã£ã«ã¿ãé©çšãããããããã«ãããããªããã¬ãŒã¹ã¯é€å€ãããŸãããã®åŸãinclusive=False
ã®ãã£ã«ã¿ãé©çšãããããããã«ããããããã¬ãŒã¹ãé€å€ãããŸããããã«ãããç¹å®ã®ã©ã€ãã©ãªãé€å€ãã€ã€ãèªåã®ã³ãŒãã®ã¿ã察象ã«ããããšãã£ãæè»ãªãã£ã«ã¿ãªã³ã°ãå¯èœã§ããð
倧èŠæš¡ãªã¢ããªã±ãŒã·ã§ã³ã§ã¯ãå€ãã®ã©ã€ãã©ãªãå éšçã«ã¡ã¢ãªã確ä¿ããŸããã¡ã¢ãªãªãŒã¯ã®åå ãç¹å®ããéã«ã¯ããŸãèªåã®ã¢ããªã±ãŒã·ã§ã³ã³ãŒãã«çŠç¹ãåœãŠãããã«ãå€éšã©ã€ãã©ãªãé€å€ãããã£ã«ã¿ãéåžžã«åœ¹ç«ã¡ãŸãã
5. ãªããžã§ã¯ãã®ãã¬ãŒã¹ããã¯ãååŸ
ç¹å®ã®ãªããžã§ã¯ããã©ãã§å²ãåœãŠãããã®ãããã³ãã€ã³ãã§ç¥ãããå ŽåããããŸããtracemalloc.get_object_traceback(obj)
é¢æ°ã䜿ããšãæå®ãããªããžã§ã¯ã obj
ãå²ãåœãŠãããéã®ãã¬ãŒã¹ããã¯ãååŸã§ããŸãã
ããã¯ãäºæããã¡ã¢ãªäžã«æ®ãç¶ããŠãããªããžã§ã¯ããçºèŠãããã®çæå ãçªãæ¢ããã®ã«åœ¹ç«ã¡ãŸãã
import tracemalloc
import gc
class MyObject:
def __init__(self, name):
self.name = name
self.data = list(range(100)) # ããããã¡ã¢ãªãæ¶è²»
tracemalloc.start(5)
# ãªããžã§ã¯ããäœæ
obj1 = MyObject("Object One")
obj2 = MyObject("Object Two")
# obj1ãžã®åç
§ãããã€ãäœæ
ref1 = obj1
ref_list = [obj1, obj2]
# obj1 ã®ãã¬ãŒã¹ããã¯ãååŸããŠã¿ã
tb1 = tracemalloc.get_object_traceback(obj1)
if tb1:
print(f"--- Traceback for obj1 ({obj1.name}) ---")
for line in tb1.format():
print(line)
else:
print("obj1 ã®ãã¬ãŒã¹ããã¯ã¯èŠã€ãããŸããã§ãã (ãŸã ãã¬ãŒã¹ãããŠããªãããã¢ãžã¥ãŒã«ãéã¢ã¯ãã£ã)ã")
# obj2 ã®ãã¬ãŒã¹ããã¯ãååŸ
tb2 = tracemalloc.get_object_traceback(obj2)
if tb2:
print(f"\n--- Traceback for obj2 ({obj2.name}) ---")
for line in tb2.format():
print(line)
# äžèŠã«ãªã£ãåç
§ãåé€ããŠã¿ã
del obj1
del ref1
ref_list.pop(0)
# ã¬ããŒãžã³ã¬ã¯ã·ã§ã³ã匷å¶å®è¡ïŒéåžžã¯äžèŠã ããã¢ã®ããïŒ
gc.collect()
# å床 obj1 ã®ãã¬ãŒã¹ããã¯ãè©Šã¿ãïŒéåžžã¯Noneã«ãªãã¯ãïŒ
# 泚æ: CPythonã®å®è£
詳现ã«ãããããã«ã¡ã¢ãªãåå©çšããããšã¯éããªã
obj1_addr = id(ref_list[0]) # obj2ã®ã¢ãã¬ã¹
print(f"\nobj2ã®ã¢ãã¬ã¹: {obj1_addr}")
# ååšããªããªããžã§ã¯ãããã¬ãŒã¹ãããŠããªããªããžã§ã¯ããæž¡ããšNoneãè¿ã
tb_nonexistent = tracemalloc.get_object_traceback(12345) # intãªããžã§ã¯ã
print(f"\nååšããªããªããžã§ã¯ãã®ãã¬ãŒã¹ããã¯: {tb_nonexistent}")
tracemalloc.stop()
get_object_traceback()
ã¯ãtracemalloc
ãã¢ã¯ãã£ãã§ããã€å¯Ÿè±¡ãªããžã§ã¯ãã®å²ãåœãŠããã¬ãŒã¹ãããŠããå Žåã«ã®ã¿ Traceback
ãªããžã§ã¯ããè¿ããŸããããã§ãªãå Žå㯠None
ãè¿ããŸãã
ãã®æ©èœã¯ãgc.get_referrers()
ïŒãªããžã§ã¯ããåç
§ããŠããä»ã®ãªããžã§ã¯ããååŸïŒã objgraph
ã®ãããªå€éšã©ã€ãã©ãªãšçµã¿åããããšãã¡ã¢ãªãªãŒã¯ã®åå ãšãªã£ãŠããåç
§é¢ä¿ãšããã®ãªããžã§ã¯ããçæãããå Žæã®äž¡æ¹ãç¹å®ããã®ã«éåžžã«åŒ·åã§ããðª
6. ç°å¢å€æ° PYTHONTRACEMALLOC ã«ããèµ·åæãã¬ãŒã¹
ããã°ã©ã ã®éåžžã«æ©ã段éããããã¯èµ·åçŽåŸããã¡ã¢ãªã¢ãã±ãŒã·ã§ã³ããã¬ãŒã¹ãããå ŽåããããŸããäŸãã°ãã€ã³ããŒãæã®ã¡ã¢ãªäœ¿çšéãã°ããŒãã«å€æ°ã®åæåãªã©ã調æ»ãããã±ãŒã¹ã§ãã
ãã®ãããªå ŽåãPythonã®ã³ãŒãå
㧠tracemalloc.start()
ãåŒã³åºããããåã«ãã¬ãŒã¹ãéå§ããå¿
èŠããããŸãããããå®çŸããã®ãç°å¢å€æ° PYTHONTRACEMALLOC
ã§ãã
ãã®ç°å¢å€æ°ãèšå®ããŠPythonããã»ã¹ãèµ·åãããšãtracemalloc
ã¢ãžã¥ãŒã«ãèªåçã«ãã¬ãŒã¹ãéå§ããŸãã
PYTHONTRACEMALLOC=1
: ããã©ã«ãã®ãã¬ãŒã æ°ïŒ1ãã¬ãŒã ïŒã§ãã¬ãŒã¹ãéå§ããŸããPYTHONTRACEMALLOC=N
(N
ã¯1以äžã®æŽæ°): æå®ãããã¬ãŒã æ°N
ã§ãã¬ãŒã¹ãéå§ããŸããäŸãã°PYTHONTRACEMALLOC=25
ãšãããšã25ãã¬ãŒã ãŸã§ä¿åããŸãã
ãŸããPython 3.7以éã§ã¯ã³ãã³ãã©ã€ã³ãªãã·ã§ã³ -X tracemalloc
ã -X tracemalloc=N
ãå©çšã§ããŸãã
䜿ãæ¹ïŒã¿ãŒããã«ïŒ:
# ç°å¢å€æ°ãèšå®ããŠPythonã¹ã¯ãªãããå®è¡ (äŸ: 10ãã¬ãŒã ã§ãã¬ãŒã¹)
export PYTHONTRACEMALLOC=10
python your_script.py
# ãŸãã¯ã³ãã³ãã©ã€ã³ãªãã·ã§ã³ã§å®è¡ (Python 3.7+)
python -X tracemalloc=10 your_script.py
ã¹ã¯ãªããå
ã§ã¯ãtracemalloc.is_tracing()
ã§ãã¬ãŒã¹ãæå¹ã«ãªã£ãŠããã確èªã§ããŸãã
# your_script.py
import tracemalloc
import time
import sys
if tracemalloc.is_tracing():
print(f"tracemalloc ã¯èµ·åæããæå¹ã§ãããã¬ãŒã æ°: {tracemalloc.get_traceback_limit()}")
else:
print("tracemalloc ã¯ãŸã æå¹ã«ãªã£ãŠããŸãããstart()ãåŒã³åºããŸãã")
tracemalloc.start()
# ã°ããŒãã«å€æ°ãªã©ãèµ·åçŽåŸã®å²ãåœãŠäŸ
startup_list = [i for i in range(1000)]
def main():
print("\nã¡ã€ã³åŠçéå§...")
local_list = [str(x) for x in range(5000)]
time.sleep(0.5)
print("ã¡ã€ã³åŠççµäº...")
if __name__ == "__main__":
main()
if tracemalloc.is_tracing():
snapshot = tracemalloc.take_snapshot()
print("\nçµäºæã®ã¡ã¢ãªçµ±èš (äžäœ5件):")
top_stats = snapshot.statistics('lineno')
for stat in top_stats[:5]:
print(stat)
tracemalloc.stop()
else:
print("\nãã¬ãŒã¹ãæå¹ã§ãªããããçµ±èšã¯ååŸã§ããŸããã")
ãã®æ¹æ³ã䜿ããšãimport
æã«ããã¢ãžã¥ãŒã«ã®ããŒãããã¯ã©ã¹å®çŸ©ãã°ããŒãã«ã¹ã³ãŒãã§ã®å€æ°åæåãªã©ãif __name__ == "__main__":
ãããã¯ãããåã«çºçããã¡ã¢ãªã¢ãã±ãŒã·ã§ã³ãææã§ããŸããããã¯ãèµ·åæã®ã¡ã¢ãªãããããªã³ããåæããéã«ç¹ã«æçšã§ããð
7. ããã©ãŒãã³ã¹ãžã®åœ±é¿ãšæ³šæç¹ â ïž
tracemalloc
ã¯éåžžã«åŒ·åãªãããã°ããŒã«ã§ãããæå¹ã«ãããšããã©ãŒãã³ã¹ïŒCPUæéãšã¡ã¢ãªäœ¿çšéïŒã«åœ±é¿ãäžããŸãããã¬ãŒã¹äžã¯ãPythonã®ã¡ã¢ãªã¢ãã±ãŒã¿ã«é¢æ°ãããã¯ãããã¢ãã±ãŒã·ã§ã³ããšã«ãã¬ãŒã¹ããã¯æ
å ±ãèšé²ã»ä¿åãããããã§ãã
- CPUãªãŒããŒããã: ã¡ã¢ãªå²ãåœãŠã®é »åºŠãé«ãã¢ããªã±ãŒã·ã§ã³ã§ã¯ã
tracemalloc
ãæå¹ã«ãããšå®è¡é床ãé ããªãå¯èœæ§ããããŸããå ¬åŒããã¥ã¡ã³ãã«ã¯å ·äœçãªæ°å€ã¯æèšãããŠããŸããããã¢ããªã±ãŒã·ã§ã³ã«ãã£ãŠã¯æ°%ããæ°å%ã®é床äœäžãèŠãããå ŽåããããŸããç¹ã«ãä¿åãããã¬ãŒã æ°ïŒnframe
ïŒãå¢ãããšããªãŒããŒãããã¯å¢å ããŸãã - ã¡ã¢ãªãªãŒããŒããã:
tracemalloc
èªäœããã¬ãŒã¹æ å ±ãä¿åããããã«ã¡ã¢ãªãæ¶è²»ããŸããä¿åãããã¬ãŒã æ°ããããã°ã©ã å šäœã®ã¡ã¢ãªã¢ãã±ãŒã·ã§ã³æ°ã«æ¯äŸããŠããã®ãªãŒããŒãããã¯å¢å ããŸããtracemalloc.get_tracemalloc_memory()
é¢æ°ã䜿ããšãtracemalloc
ã¢ãžã¥ãŒã«èªèº«ã䜿çšããŠããã¡ã¢ãªéã確èªã§ããŸãã
import tracemalloc
import time
tracemalloc.start(25) # ãã¬ãŒã æ°ã25ã«èšå®
# 倧éã®å°ããªãªããžã§ã¯ããçæããŠã¿ã
start_time = time.perf_counter()
for _ in range(100000):
a = object()
end_time = time.perf_counter()
print(f"åŠçæé (tracemallocæå¹): {end_time - start_time:.4f} ç§")
# tracemallocã䜿çšããŠããã¡ã¢ãªéã確èª
mem_usage = tracemalloc.get_tracemalloc_memory()
print(f"tracemalloc ã®ã¡ã¢ãªäœ¿çšé: {mem_usage / 1024:.2f} KiB")
tracemalloc.stop()
# æ¯èŒã®ããã« tracemalloc ç¡å¹ã§åãåŠçãå®è¡
start_time_no_trace = time.perf_counter()
for _ in range(100000):
a = object()
end_time_no_trace = time.perf_counter()
print(f"\nåŠçæé (tracemallocç¡å¹): {end_time_no_trace - start_time_no_trace:.4f} ç§")
ãã®ã³ãŒããå®è¡ãããšãtracemalloc
ãæå¹ãªå Žåã®æ¹ãåŠçæéãé·ããªãããšã確èªã§ããã§ãããïŒå·®ã¯ç°å¢ãPythonããŒãžã§ã³ã«ãããŸãïŒã
泚æç¹ãŸãšã:
tracemalloc
ã¯ãããã°ããããã¡ã€ãªã³ã°ã®ããã®ããŒã«ã§ãããæ¬çªç°å¢ã§åžžã«æå¹ã«ããŠããã¹ãã§ã¯ãããŸãããããã©ãŒãã³ã¹ãžã®åœ±é¿ã倧ããããã§ãã- ã¡ã¢ãªãªãŒã¯èª¿æ»ãã¡ã¢ãªäœ¿çšéã®æé©åãè¡ãéçºã»ãã¹ãç°å¢ã§ã®ã¿äœ¿çšããããšãæšå¥šããŸãã
- ä¿åãããã¬ãŒã æ° (
nframe
) ã¯ãå¿ èŠãªæ å ±ãåŸãããæå°éã®æ°ã«èšå®ããããšã§ããªãŒããŒããããæããããšãã§ããŸããå€ãã®å Žåãããã©ã«ãã®1ããæ°ãã¬ãŒã çšåºŠã§ååãªå Žåãå€ãã§ãã tracemalloc.get_tracemalloc_memory()
ã§ããŒã«èªäœã®ã¡ã¢ãªæ¶è²»éãç£èŠããéå°ã«ãªã£ãŠããªãã確èªããŸãããã- Cæ¡åŒµã«ãã£ãŠç¢ºä¿ãããã¡ã¢ãªã¯è¿œè·¡ã§ããªãç¹ãç解ããŠããå¿ èŠããããŸãã
tracemalloc
ãè³¢ãå©çšããããšãéèŠã§ããæ¬çªç°å¢ã§ã®ã¡ã¢ãªç£èŠã«ã¯ããããªãŒããŒãããã®å°ãªãå¥ã®ææ³ïŒäŸ: OSã¬ãã«ã§ã®ç£èŠããµã³ããªã³ã°ããŒã¹ã®ãããã¡ã€ã©ãªã©ïŒãæ€èšããããšãæå¹ã§ãã
8. å®è·µçãªãŠãŒã¹ã±ãŒã¹ ð
tracemalloc
ã¯æ§ã
ãªã·ããªãªã§åœ¹ç«ã¡ãŸãã
-
Webã¢ããªã±ãŒã·ã§ã³ (Flask, Djangoãªã©):
- ãªã¯ãšã¹ãåŠçäžã«ã¡ã¢ãªäœ¿çšéãç°åžžã«å¢å ããŠããªãã調æ»ã
- ç¹å®ã®ãšã³ããã€ã³ãããã¥ãŒé¢æ°ãã¡ã¢ãªãªãŒã¯ã®åå ã«ãªã£ãŠããªããç¹å®ã
- ãªã¯ãšã¹ãååŸã§ã¹ãããã·ã§ãããæ¯èŒãããªãŒã¯ç®æãç¹å®ã
- é·æé皌åãããWebãµãŒããŒããã»ã¹å šäœã®ã¡ã¢ãªå¢å åŸåãåæã
-
é·æéå®è¡ããããããåŠçã»ããŒã¿åŠç:
- åŠçãé²ãã«ã€ããŠã¡ã¢ãªäœ¿çšéãå¢ãç¶ããæçµçã«ã¯ã©ãã·ã¥ããåé¡ã調æ»ã
- ã«ãŒãåŠçã®åã€ãã¬ãŒã·ã§ã³ååŸã§ã¹ãããã·ã§ãããæ¯èŒããã¡ã¢ãªãé©åã«è§£æŸãããŠããã確èªã
- 倧èŠæš¡ãªããŒã¿ã»ãããåŠçããéã«ãã©ã®ããŒã¿æ§é ãåŠçã¹ããããã¡ã¢ãªã倧éã«æ¶è²»ããŠãããç¹å®ããæé©åïŒäŸ: ãžã§ãã¬ãŒã¿ã®äœ¿çšããã£ã³ã¯åŠçïŒã
-
ã©ã€ãã©ãªããã¬ãŒã ã¯ãŒã¯ã®éçº:
- éçºäžã®ã©ã€ãã©ãªãæå³ããã¡ã¢ãªãªãŒã¯ãåŒãèµ·ãããŠããªãããã¹ãã
- å éšããŒã¿æ§é ã®ã¡ã¢ãªå¹çãè©äŸ¡ã»æ¹åã
-
ãã¹ãã³ãŒãã§ã®å©çš:
- ç¹å®ã®ãã¹ãã±ãŒã¹å®è¡ååŸã§ã¡ã¢ãªäœ¿çšéãæ¯èŒãããã¹ã察象ã®ã³ãŒããã¡ã¢ãªãªãŒã¯ãèµ·ãããªãããšã確èªããã¢ãµãŒã·ã§ã³ãè¿œå ã
äŸãã°ãFlaskã¢ããªã±ãŒã·ã§ã³ã§ç¹å®ã®ãªã¯ãšã¹ããã³ãã©ã«ãããã¡ã¢ãªäœ¿çšéã調æ»ããå Žåã以äžã®ãããªãã³ã¬ãŒã¿ãäœæããŠé©çšããããšãèããããŸãïŒç°¡ç¥åããäŸïŒã
from functools import wraps
import tracemalloc
from flask import Flask, request
app = Flask(__name__)
# 泚æ: æ¬çªç°å¢ã§ã®äœ¿çšã¯éæšå¥šããããã°ç®çã®ã¿ã
# ãŸããè€æ°ãªã¯ãšã¹ããåæã«åŠçããå Žåãã¹ãããã·ã§ãããæ··ããå¯èœæ§ãããããã
# å®éã®å©çšã§ã¯ãã泚ææ·±ãå®è£
ãå¿
èŠã§ãïŒäŸïŒãªã¯ãšã¹ãããšã«äžæãªIDã§ç®¡çïŒã
def trace_request_memory(f):
@wraps(f)
def decorated_function(*args, **kwargs):
if not tracemalloc.is_tracing():
tracemalloc.start() # ãã¬ãŒã¹ãéå§ãããŠããªããã°éå§
snapshot1 = tracemalloc.take_snapshot()
try:
result = f(*args, **kwargs)
finally:
snapshot2 = tracemalloc.take_snapshot()
top_stats = snapshot2.compare_to(snapshot1, 'lineno')
print(f"\n--- Request to {request.path} Memory Diff ---")
if top_stats:
print("[Top 5 Memory Differences]")
for stat in top_stats[:5]:
print(stat)
else:
print("No significant memory difference detected.")
print("-" * 40)
return result
return decorated_function
@app.route('/process_large_data')
@trace_request_memory
def process_large_data():
# ãããšã¡ã¢ãªãå€ã䜿ãåŠç
data_size = request.args.get('size', default=10000, type=int)
large_list = ['item ' * 100] * data_size
# ããã§äœããã®åŠçãè¡ã
processed_count = len(large_list)
# large_list ã解æŸããã«è¿ãïŒãªãŒã¯ã®ã·ãã¥ã¬ãŒã·ã§ã³ïŒ
# æ¬æ¥ã¯é©åã«è§£æŸãã¹ã
return f"Processed {processed_count} items."
@app.route('/')
def index():
return "Hello! Try /process_large_data?size=50000"
if __name__ == '__main__':
# éçºãµãŒããŒã§å®è¡ãPYTHONTRACEMALLOCã¯èšå®ããªãæ³å®ã
app.run(debug=True, use_reloader=False) # reloaderã¯ãããã°ã«åœ±é¿ããããšããã
ãã®äŸã§ã¯ã/process_large_data
ãšã³ããã€ã³ããžã®ãªã¯ãšã¹ãååŸã§ã¹ãããã·ã§ãããååŸããã³ã³ãœãŒã«ã«å·®åãåºåããŸããããã«ããããªã¯ãšã¹ãåŠçã«ããã¡ã¢ãªå¢å ãç°¡åã«ç¢ºèªã§ããŸãããã ããäžèšã³ãŒãã¯ç°¡ç¥åãããŠãããã¹ã¬ããã»ãŒããã£ãªã©ã®èæ
®ã¯ãããŠããŸãããå®éã®ã¢ããªã±ãŒã·ã§ã³ã«çµã¿èŸŒãå Žåã¯æ³šæãå¿
èŠã§ãã
9. ãŸãšã âš
tracemalloc
ã¯ãPythonã¢ããªã±ãŒã·ã§ã³ã«ãããã¡ã¢ãªé¢é£ã®åé¡ã蚺æããããã®éåžžã«åŒ·åã§äŸ¿å©ãªæšæºã©ã€ãã©ãªã§ãã
- ç°¡åãªéå§:
start()
/stop()
ã§ç°¡åã«ãã¬ãŒã¹ãéå§ã»åæ¢ã§ããŸãã - ã¹ãããã·ã§ããæ¯èŒ:
take_snapshot()
ãšcompare_to()
ã§ã¡ã¢ãªãªãŒã¯ã®åå ç®æãç¹å®ã§ããŸãã - 詳现ãªçµ±èšæ
å ±:
statistics()
ã§ãã©ã®ãã¡ã€ã«ãè¡ãã¡ã¢ãªãå€ãæ¶è²»ããŠããã詳现ã«åæã§ããŸãã - ãã£ã«ã¿ãªã³ã°:
Filter
ã¯ã©ã¹ã§ãã€ãºãé€å»ãã調æ»å¯Ÿè±¡ãçµã蟌ããŸãã - ãªããžã§ã¯ã远跡:
get_object_traceback()
ã§ç¹å®ã®ãªããžã§ã¯ããã©ãã§çæãããã远跡ã§ããŸãã - èµ·åæãã¬ãŒã¹:
PYTHONTRACEMALLOC
ç°å¢å€æ°ã§ããã°ã©ã éå§çŽåŸãããã¬ãŒã¹ã§ããŸãã
äžæ¹ã§ãããã©ãŒãã³ã¹ãžã®åœ±é¿ããããããæ¬çªç°å¢ã§ã®åžžæå©çšã¯é¿ããéçºã»ãã¹ããã§ãŒãºã§ã®å©çšã«çããã¹ãã§ãã
Pythonã¢ããªã±ãŒã·ã§ã³ã§ã¡ã¢ãªäœ¿çšéãæ°ã«ãªãå Žåããåå äžæã®ã¡ã¢ãªãªãŒã¯ã«æ©ãŸãããŠããå Žåã¯ããã² tracemalloc
ã掻çšããŠã¿ãŠãã ããããã£ãšåé¡è§£æ±ºã®å€§ããªå©ããšãªãã¯ãã§ãïŒð Happy Memory Debugging! ð
ã³ã¡ã³ã