embulk

Embulk is an Open-source Pluggable Bulk Data Loader
to/from varieties of storages, file formats, databases, cloud services, and else.

Latest Versions

Stable
v0.11.5
Note that v0.11 is not fully compatible with v0.9. Several plugins for v0.9 would not work with v0.11.
Take a look at this article for details.
Past stable
v0.9.25

Recent Articles

Embulk is a bulk data loader. It helps data transfer between types of databases, storages, file formats, cloud services, and else.

Embulk supports:

  • Combination of input and output from varieties of plugins
  • Plugins released in Maven and Ruby gem repositories
  • Automated guess of input file formats
  • Parallel execution to deal with big data sets
  • Transaction control to guarantee all-or-nothing

Community

For Users
User Forum: GitHub Discussions
For Developers
Chat: Zulip at https://embulk-dev.zulipchat.com/
Developer Resources: https://dev.embulk.org/

Quick Start

Set up the Java Runtime Environment. Embulk runs on Java. Embulk v0.11 officially supports only Java 8, but expected to work somehow with Java 11, 17, and 21.

Download the latest `embulk-X.Y.Z.jar` from: [GitHub Releases](https://github.com/embulk/embulk/releases)

Try the following commands. embulk example command generates an example CSV file.

java -jar embulk-X.Y.Z.jar example ./try1
java -jar embulk-X.Y.Z.jar guess ./try1/seed.yml -o config.yml
java -jar embulk-X.Y.Z.jar preview config.yml
java -jar embulk-X.Y.Z.jar run config.yml

Install an Embulk plugin

Install an Embulk plugin (ex. S3 Input Plugin) with the following command.

java -jar embulk-X.Y.Z.jar install "org.embulk:embulk-input-s3:0.6.0"

Set up with JRuby

If you need to use a RubyGems-style Embulk plugin, you'll have to set up with JRuby.

Download your preferred version 9.x.y of JRuby Complete jar from JRuby Downloads, and put it in your favorite directory.

Embulk is tested with old JRuby 9.1.15.0 as of now. (However, older JRuby has an issue in installing Ruby gems. You may need to try newer JRuby.)

Then, create ~/.embulk/embulk.properties as follows.

jruby=file:///path/to/jruby-complete-9.x.y.z.jar

Finally, install some Ruby gems.

embulk gem install embulk -v 0.11.5 # Must be the same Embulk version
embulk gem install msgpack
embulk gem install bundler # if you need Bundler
embulk gem install liquid  # if you need Liquid

Presentation