DEV Community

vic
vic

Posted on

A simple log collector listening log file written to clickhouse

A simple log collector, Monitor files through tail -F command, write to the Clickhouse. Very low occupancy of resources, It can process more than 100 thousand log information per second

install

composer require lizhichao/log2ck

example

tail -F apapche/access.log | php apache_log.php -h tcp://127.0.0.1:9000 -u default -p 123456 -d logs -t apache_log

options :

  • -h clickhouse host port
  • -u clickhouse user name
  • -p clickhouse password
  • -d clickhouse database name
  • -t clickhouse table name

principle

Use clickhouse to support streaming writing

Code factory library used

https://github.com/lizhichao/log2ck

https://github.com/lizhichao/one-ck

Discussion (0)