return specific fields in search (json)

[wdweng]

Bug

Like in your essay, you says y-slope can store and compress each table in a columnar manner that
maximizes the compression ratio and search speed
but when i uses the search feature, it just returns all log lines that meets the requirement

CLP version

3.0

Environment

ubuntu 22.04 lastest docker

Reproduction steps

search

[gibber9809]

Hello @wdweng,

The clp-json subsystem has support for columnar search and projection on JSON logs while the clp-text subsystem is designed for classic log-text. μSlope from our paper corresponds to this clp-json subsystem. If columnar search doesn't seem to be working a good first step would be to check that you're using clp-json if you're using the package, or the clp-s binary if you're using the command line interface directly.

If you need a reference for the search syntax that we support for in clp-json/clp-s you can find documentation here.

By default search will return the entirety of each matching record, but if you're interested in retrieving only some of the columns we do have support for projection, though projection support is currently isn't exposed in the clp-json package.

If you're using the command line interface you can specify the list of projected columns at the end of your command e.g.

clp-s s <archive> <kql query> --projection column1 column2 ...

you can see a bit more information about --projection and other search options by running clp-s s --help.

If you're interested in a more classic SQL interface for columnar search and projection we're also currently integrating with presto/velox. That integration is still being open sourced, but should hopefully become available in the next couple of months. The RFC for that integration is available here if you're curious.

I hope that answers some of your questions.