▲ 9 r/Common_Lisp
How to improve this terribly slow reading?
I was reading TaDa and want to read the lineitem.csv example in Common Lisp, so I wrote:
(with-open-file (stream file)
(loop :for line := (read-line stream nil) :while line
:summing (parse-integer (nth 4 (str:split "|" line)))))
However, on my computer, it took about 17sec to run!
Maybe it was my fault, so I change to cl-csv:
(let ((sum 0))
(cl-csv:do-csv (row #P"lineitem.csv" :separator #\|)
(incf sum (parse-integer (nth 4 row))))
sum)
and it is even worse...
Profiling shows that
Self Total Cumul
Nr Count % Count % Count % Calls Function
------------------------------------------------------------------------
1 9903 80.0 9903 80.0 9903 80.0 - foreign function read
2 414 3.3 414 3.3 10317 83.3 - foreign function __semwait_signal
3 218 1.8 1878 15.2 10535 85.1 - cl-ppcre:split
4 182 1.5 404 3.3 10717 86.5 - (lambda (string cl-ppcre::start cl-ppcre::end) :in cl-ppcre::create-scanner-aux)
5 147 1.2 820 6.6 10864 87.7 - (sb-pcl::emf cl-ppcre:scan)
6 131 1.1 159 1.3 10995 88.8 - sb-pcl::check-applicable-keywords
7 122 1.0 122 1.0 11117 89.8 - (lambda (cl-ppcre::start-pos) :in cl-ppcre::create-char-searcher)
8 98 0.8 509 4.1 11215 90.6 - (sb-pcl::fast-method cl-ppcre:scan (function t))
9 88 0.7 103 0.8 11303 91.3 - (lambda (sb-pcl::.arg0. sb-pcl::.arg1. &rest sb-pcl::.rest.) :in "SYS:SRC;PCL;DLISP3.LISP")
10 83 0.7 121 1.0 11386 91.9 - sb-kernel:ub32-bash-copy
This is really wired.
Notes: I was running SBCL 2.6.3 on macOS 26 (MacBook Air M2). The lineitem.csv (~=750MB) is generated via (duckdb):
.bail on
ATTACH IF NOT EXISTS ':memory:';
USE memory;
INSTALL tpch;
LOAD tpch;
CALL dbgen(sf = 1);
.shell rm -f lineitem.csv
COPY (SELECT * REPLACE (l_quantity :: bigint AS l_quantity)
FROM lineitem)
TO 'lineitem.csv' (delimiter '|', header false);
u/lucky_magick — 6 days ago