Babelfish logo

Babelfish

  •  0 ratings
In category: Software Development / IDE & Tools

About Babelfish

Self-hosted server for source code parsing. It can parse any file, in any supported language, extract an Abstract Syntax Tree from it, and convert it to a Universal Abstract Syntax Tree which can enable further analysis and transformation.

  •   471  
  •   0  
  •   0  
  •   0  
Github stats:
  •  Commits: N/A  
  •   N/A  
  •   N/A  
  •  Latest commit: N/A  

Deploy this app to Linode with a free $100 credit!

Languages/Platforms/Technologies:
Lincenses:

More about Babelfish

bblfshd Build Status codecov license GitHub release

This repository contains bblfsh daemon (bblfshd), which includes the runtime that runs the driver in containers and the bblfshctl, a cli tool used to control the installed drivers and query the status of the daemon.

Drivers are implemented as Docker images, each having their own repository in the bblfsh organization on GitHub. For more information, see bblfsh SDK documentation.

Getting Started

See the Getting Started guide.

Quick start

This project is now part of source{d} Engine, which provides the simplest way to get started with a single command. Visit sourced.tech/engine for more information.

Rootless mode

The recommended way to run bblfshd by itself is using Docker:

docker run -d --name bblfshd \
  -p 9432:9432 \
  -v /var/lib/bblfshd:/var/lib/bblfshd \
  -v /proc:/newproc \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd

On macOS, use this command instead to use a Docker volume:

docker run -d --name bblfshd \
  -p 9432:9432 \
  -v bblfsh-storage:/var/lib/bblfshd bblfsh/bblfshd \
  -v /proc:/newproc \
  --security-opt seccomp=./bblfshd-seccomp.json \
  bblfsh/bblfshd

To understand the flags -v /proc:/newproc and --security-opt seccomp=./bblfshd-seccomp.json, where bblfshd-seccomp.json is a file present in this repo, and check further requirements, please refer to rootless.md. bblfshd is based on container technology and interacts with the kernel at a low level. It exposes a gRPC server at the port 9432 by default which is used by the clients to interact with the server. Also, we mount the path /var/lib/bblfshd/ where all the driver images and container instances will be stored.

Privileged mode

We advise against it, but if you prefer to run bblfshd in privileged mode to skip configuration steps of rootless.md, you could do, in Linux:

docker run -d --name bblfshd --privileged -p 9432:9432 -v /var/lib/bblfshd:/var/lib/bblfshd bblfsh/bblfshd

or macOs:

docker run -d --name bblfshd --privileged -p 9432:9432 -v bblfsh-storage:/var/lib/bblfshd bblfsh/bblfshd

Install drivers

Now you need to install the driver images into the daemon, you can install the official images just running the command:

docker exec -it bblfshd bblfshctl driver install --all

You can check the installed versions by executing:

docker exec -it bblfshd bblfshctl driver list

+----------+-------------------------------+---------+--------+---------+-----+-------------+
| LANGUAGE |             IMAGE             | VERSION | STATUS | CREATED | GO  |   NATIVE    |
+----------+-------------------------------+---------+--------+---------+-----+-------------+
| python   | //bblfsh/python-driver:latest | v1.1.5  | beta   | 4 days  | 1.8 | 3.6.2       |
| java     | //bblfsh/java-driver:latest   | v1.1.0  | alpha  | 6 days  | 1.8 | 8.131.11-r2 |
+----------+-------------------------------+---------+--------+---------+-----+-------------+

To test the driver you can execute a parse request to the server with the bblfshctl parse command, and an example contained in the Docker image:

docker exec -it bblfshd bblfshctl parse /opt/bblfsh/etc/examples/python.py

SELinux

If your system has SELinux enabled (which is the default in Fedora, Red Hat, CentOS and many others) you'll need to compile and load a policy module before running the bblfshd Docker image or running driver containers will fail with a permission denied message in the logs.

To do this, run these commands from the project root:

cd selinux/
sh compile.sh
semodule -i bblfshd.pp

If you were already running an instance of bblfshd, you will need to delete the container (docker rm -f bblfshd) and run it again (docker run...).

Once the module has been loaded with semodule the change should persist even if you reboot. If you want to permanently remove this module run semodule -d bblfshd.

Alternatively, you could set SELinux to permissive module with:

echo 1 > /sys/fs/selinux/enforce

(doing this on production systems which usually have SELinux enabled by default should be strongly discouraged).

Development

If you wish to work on bblfshd , you'll first need Go installed on your machine (version 1.11+ is required) and Docker. Docker is used to build and run tests in an isolated environment.

For local development of bblfshd, first make sure Go is properly installed and that a GOPATH has been set. You will also need to add $GOPATH/bin to your $PATH.

Next, using Git, clone this repository into $GOPATH/src/github.com/bblfsh/bblfshd. All the necessary dependencies are automatically installed, so you just need to type make. This will compile the code and then run the tests. If this exits with exit status 0, then everything is working!

Dependencies

Ensure you have ostree and development libraries for ostree installed.

You can install from your distribution pack manager as follow, or built from source (more on that here).

Debian, Ubuntu, and related distributions:

$ apt-get install libostree-dev

Fedora, CentOS, RHEL, and related distributions:

$ yum install -y ostree-devel

Arch and related distributions:

$ pacman -S ostree

Building From Source

Build with:

$ make build

Running Tests

Run tests with:

$ make test

Environment variables

  • BBLFSHD_MAX_DRIVER_INSTANCES - maximal number of driver instances for each language. Default to a number of CPUs.

  • BBLFSHD_MIN_DRIVER_INSTANCES - minimal number of driver instances that will be run for each language. Default to 1.

Enable tracing

Bblfshd supports OpenTracing that can be used to profile request on a high level or trace individual requests to bblfshd and/or language drivers.

To enable it, you can use Jaeger. The easiest way is to start all-in-one Jaeger image:

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.8

For Docker installation of bblfshd add the following flags:

--link jaeger:jaeger -e JAEGER_AGENT_HOST=jaeger -e JAEGER_AGENT_PORT=6831 -e JAEGER_SAMPLER_TYPE=const -e JAEGER_SAMPLER_PARAM=1

For bblfshd running locally, set following environment variables:

JAEGER_AGENT_HOST=localhost JAEGER_AGENT_PORT=6831 JAEGER_SAMPLER_TYPE=const JAEGER_SAMPLER_PARAM=1

Run few requests, and check traces at http://localhost:16686.

For enabling tracing in production, consult Jaeger documentation.

License

GPLv3, see LICENSE

Comments (0)

Please login to join the discussion on this project.

Babelfish Reviews (0)

Overall Rating

None

based on 0 ratings

Please login to review this project.

No reviews for this project yet.

↑ back to top

pCloud Lifetime

Popular Projects

FluxBB

in Social Networks and Forums
 33k    0    0    0  

Nextcloud

in File Transfer & Synchronization
 20k    1    1    0  

Libreddit

in Social Networks and Forums
 7k    0    1    0  

Dashboard

in Personal Dashboards
 6k    0    0    0  

Audiobookshelf

in Audio Streaming
 5k    0    1    0  

CasaOS

in Self-hosting Solutions
 5k    0    0    0  

Mediagoblin

in Photo and Video Galleries
 4k    0    0    0  

Most Discussed

Nextcloud

in File Transfer & Synchronization
 20k    1    1    0  

Tube Archivist

in Automation
 3k    0    1    0  

OneDev

in Project Management
 2k    0    0    0  

iodine

in Proxy
 2k    0    0    0  

Alf.io

in Booking and Scheduling
 2k    0    0    0  

sysPass

in Password Managers
 1k    0    0    0  

Misskey

in Social Networks and Forums
 2k    0    0    0  
Linux VPS from $11/yr.
RackNerd VPS for $11.38/mo

Top Rated Projects

Gitea

 1 rating
in Project Management

Bagisto

 1 rating
in E-commerce

LinkAce

 1 rating
in Bookmarks and Link Sharing

Pydio

 1 rating
in File Transfer & Synchronization

Audiobookshelf

 1 rating
in Audio Streaming

Nextcloud

 1 rating
in File Transfer & Synchronization

Seafile

 1 rating
in File Transfer & Synchronization

Categories

You May Also Be Interested In

HttPlaceholder logo
HttPlaceholder cover

HttPlaceholder

Quickly mock away any webservice using HttPlaceholder. HttP…

Eclipse Che logo
Eclipse Che cover

Eclipse Che

Open source workspace server and cloud IDE.

Wakapi logo
Wakapi cover

Wakapi

Tracking tool for coding statistics, compatible with WakaTi…