sist2
- 0 ratings
Lightning-fast file system indexer and search tool.
Deploy this app to Linode with a free $100 credit!
Demo: sist2.simon987.net
sist2 (Simple incremental search tool)
Warning: sist2 is in early development
* See format support
** See Archive files
*** See OCR
**** See Named-Entity Recognition
version: "3"
services:
elasticsearch:
image: elasticsearch:7.17.9
restart: unless-stopped
environment:
- "discovery.type=single-node"
- "ES_JAVA_OPTS=-Xms2g -Xmx2g"
sist2-admin:
image: simon987/sist2:3.0.4-x64-linux
restart: unless-stopped
volumes:
- ./sist2-admin-data/:/sist2-admin/
- /:/host
ports:
- 4090:4090 # sist2
- 8080:8080 # sist2-admin
working_dir: /root/sist2-admin/
entrypoint: python3 /root/sist2-admin/sist2_admin/app.pyNavigate to http://localhost:8080/ to configure sist2-admin.
Have an Elasticsearch (>= 6.8.X, ideally >=7.14.0) instance running
docker run -d -p 9200:9200 -e "discovery.type=single-node" elasticsearch:7.17.9Download the latest sist2 release.
Select the file corresponding to your CPU architecture and mark the binary as executable with chmod +x.
Example usage:
sist2 scan ~/Documents --output ./documents.sist2sist2 index ./documents.sist2sist2 web ./documents.sist2| File type | Library | Content | Thumbnail | Metadata |
|---|---|---|---|---|
| pdf,xps,fb2,epub | MuPDF | text+ocr | yes | author, title |
| cbz,cbr | libscan | - | yes | - |
audio/* |
ffmpeg | - | yes | ID3 tags |
video/* |
ffmpeg | - | yes | title, comment, artist |
image/* |
ffmpeg | ocr | yes | Common EXIF tags, GPS tags |
| raw, rw2, dng, cr2, crw, dcr, k25, kdc, mrw, pef, xf3, arw, sr2, srf, erf | LibRaw | no | yes | Common EXIF tags, GPS tags |
| ttf,ttc,cff,woff,fnt,otf | Freetype2 | - | yes, bmp |
Name & style |
text/plain |
libscan | yes | no | - |
| html, xml | libscan | yes | no | - |
| tar, zip, rar, 7z, ar ... | Libarchive | yes* | - | no |
| docx, xlsx, pptx | libscan | yes | if embedded | creator, modified_by, title |
| doc (MS Word 97-2003) | antiword | yes | no | author, title |
| mobi, azw, azw3 | libmobi | yes | yes | author, title |
| wpd (WordPerfect) | libwpd | yes | no | planned |
| json, jsonl, ndjson | libscan | yes | - | - |
* See Archive files
sist2 will scan files stored into archive files (zip, tar, 7z...) as if they were directly in the file system. Recursive (archives inside archives) scan is also supported.
Limitations:
.gif, .mp4 w/ fragmented metadata etc.)
is limitted (see --mem-buffer option)You can enable OCR support for ebook (pdf,xps,fb2,epub) or image file types with the
--ocr-lang <lang> option in combination with --ocr-images and/or --ocr-ebooks.
Download the language data files with your package manager (apt install tesseract-ocr-eng) or
directly from Github.
The simon987/sist2 image comes with common languages
(hin, jpn, eng, fra, rus, spa, chi_sim, deu) pre-installed.
You can use the + separator to specify multiple languages. The language
name must be identical to the *.traineddata file installed on your system
(use chi_sim rather than chi-sim).
Examples:
sist2 scan --ocr-ebooks --ocr-lang jpn ~/Books/Manga/
sist2 scan --ocr-images --ocr-lang eng ~/Images/Screenshots/
sist2 scan --ocr-ebooks --ocr-images --ocr-lang eng+chi_sim ~/Chinese-Bilingual/sist2 v3.0.4+ supports named-entity recognition (NER). Simply add a supported repository URL to Configuration > Machine learning options > Model repositories to enable it.
The text processing is done in your browser, no data is sent to any third-party services. See simon987/sist2-ner-models for more details.
| URL | Maintainer | Purpose |
|---|---|---|
| simon987/sist2-ner-models | simon987 | General |
You can compile sist2 by yourself if you don't want to use the pre-compiled binaries
git clone --recursive https://github.com/simon987/sist2/
cd sist2
docker build . -t my-sist2-image
# Copy sist2 executable from docker image
docker run --rm --entrypoint cat my-sist2-image /root/sist2 > sist2-x64-linuxapt install gcc g++ python3 yasm ragel automake autotools-dev wget libtool libssl-dev curl zip unzip tar xorg-dev libglu1-mesa-dev libxcursor-dev libxml2-dev libxinerama-dev gettext nasm git nodejsInstall vcpkg dependencies
vcpkg install curl[core,openssl] sqlite3 cpp-jwt pcre cjson brotli libarchive[core,bzip2,libxml2,lz4,lzma,lzo] pthread tesseract libxml2 libmupdf gtest mongoose libmagic libraw gumbo ffmpeg[core,avcodec,avformat,swscale,swresample]Build
git clone --recursive https://github.com/simon987/sist2/
(cd sist2-vue; npm install; npm run build)
(cd sist2-admin/frontend; npm install; npm run build)
cmake -DSIST_DEBUG=off -DCMAKE_TOOLCHAIN_FILE=<VCPKG_ROOT>/scripts/buildsystems/vcpkg.cmake .
makePlease login to review this project.
No reviews for this project yet.
Cloud-native neural search framework for any kind of data.
Private web search engine.
Privacy-respecting, hackable metasearch engine.
Comments (0)
Please login to join the discussion on this project.