Motivation: Some workloads could have crazy hash settings, or use very large networks, or non-verbatim networks, or maybe some workers just have tiny memory. It would be nice to make sure we can reject workloads that will OOM us.
Want to add --memory-limit, used with something like 12GB or 12000MB. Can yell at the user if they use this but we're not on Linux. We should probably also convey this info to the server, aligned with #279.
Basic flow: Client requests a test. Builds the engines. During benching, it tracks peak memory usage in bench.py, and returns the peak usage along with the bench and NPS values. Initial Claude mockup for that below, which I played with locally. Using the peak memory info, the Threads= settings for each engine, and the Hash= settings for each engine, we estimate the memory needed. Multiply that by 1.25x to give some overhead; and then add 1GB to cover our bases with Fastchess + Python. the estiamted memory exceeds our --memory-limit, then we throw an error back to the server. Similar to how we report bad benches or inability to compile. Client will then get a new workload, and not request the same one, as blacklisting is already present.
Should update the bench_engine.py and bench_all.py scripts in the Scripts/ directory to report this new information in some way. bench_engine.py is a great way to test this.
def sample_engine_memory(workers):
total = 0
for worker in workers:
try: # Engines are direct children of the multiprocessing workers
for engine in worker.children(recursive=True):
try:
info = engine.memory_full_info()
total += getattr(info, 'pss', info.uss)
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
except (psutil.NoSuchProcess, psutil.AccessDenied):
pass
return total
def monitor_peak_memory(worker_pids, stop_event, result):
workers = []
for pid in worker_pids:
try: workers.append(psutil.Process(pid))
except psutil.NoSuchProcess: pass
peak = 0
while not stop_event.is_set():
peak = max(peak, sample_engine_memory(workers))
time.sleep(MEMORY_SAMPLE_SECONDS)
result['peak'] = max(peak, sample_engine_memory(workers))
Motivation: Some workloads could have crazy hash settings, or use very large networks, or non-verbatim networks, or maybe some workers just have tiny memory. It would be nice to make sure we can reject workloads that will OOM us.
Want to add --memory-limit, used with something like 12GB or 12000MB. Can yell at the user if they use this but we're not on Linux. We should probably also convey this info to the server, aligned with #279.
Basic flow: Client requests a test. Builds the engines. During benching, it tracks peak memory usage in bench.py, and returns the peak usage along with the bench and NPS values. Initial Claude mockup for that below, which I played with locally. Using the peak memory info, the Threads= settings for each engine, and the Hash= settings for each engine, we estimate the memory needed. Multiply that by 1.25x to give some overhead; and then add 1GB to cover our bases with Fastchess + Python. the estiamted memory exceeds our --memory-limit, then we throw an error back to the server. Similar to how we report bad benches or inability to compile. Client will then get a new workload, and not request the same one, as blacklisting is already present.
Should update the bench_engine.py and bench_all.py scripts in the Scripts/ directory to report this new information in some way. bench_engine.py is a great way to test this.