ElasticSearch stores each piece of data in a document.
That's what I need.
Using the bulk API.
Transform the raw data file from data.json to be new_data.json .
And then do this to import data to ElasticSearch :
curl -s -XPOST 'localhost:9200/_bulk' --data-binary @new_data.json
For example, I now have a raw JSON data file as following:
The file data.json
{"key1":"valueA_row_1","key2":"valueB_row_1","key3":"valueC_row_1"}
{"key1":"valueA_row_2","key2":"valueB_row_2","key3":"valueC_row_2"}
{"key1":"valueA_row_3","key2":"valueB_row_3","key3":"valueC_row_3"}
Then I need to import these data to elasticsearch. So I have to manipulate this file by naming its index and type.
A new file will be created new_data.json
{"index":{"_index":"myindex1","_type":"mytype1"}} {"key1":"valueA_row_1","key2":"valueB_row_1","key3":"valueC_row_1"} {"index":{"_index":"myindex1","_type":"mytype1"}} {"key1":"valueA_row_2","key2":"valueB_row_2","key3":"valueC_row_2"} {"index":{"_index":"myindex1","_type":"mytype1"}} {"key1":"valueA_row_3","key2":"valueB_row_3","key3":"valueC_row_3"}
There are information above each of the data line in the file new_data.json
And if the JSON data file contains data those are not in the same _index or _type, just change the {"index":{"_******** line
Here is an example of a valid JSON file for elasticsearch.
full_data.json
{"index":{"_index":"myindex1","_type":"mytype1"}} {"key1":"value1","key2":"value2","key3":"value3"} {"index":{"_index":"myindex1","_type":"mytype1"}} {"key1":"abcde","key2":"efg","key3":"klm"} {"index":{"_index":"myindex2","_type":"mytype2"}} {"newkey":"newvalue"}
Notice that : There are 2 indexes in the file above. They are myindex1 and myindex2
And the data schema in index myindex2 is different from that in index myindex1 .
That's why it's so important to have so many lines of {"index":{"_******** in the new data file.
-----
Now I am coding a python scripe to manipulate with some raw JSON data files.
Let's assume each line of the JSON data file are in the same schema. And I will do this to generate the schema out.
example_raw_data.json
import sys def get_schema(): """ """ return None if __name__ == "__main__": print(get_schema)
-------------Updated on 27th Nov. 2015 ----------
I solved this by inventing a new wheel
You can check this out:
https://github.com/xros/json-py-es
-------------Updated on 28th Nov. 2015 at 01:33 A.M. ----------
pip install jsonpyes
I wrote this module and it works!
Happy hacking!