performance: gjson parsing for readXLMeta, listParts, getObjectInfo. (#2631)
- Using gjson for constructing xlMetaV1{} in realXLMeta.
- Test for parsing constructing xlMetaV1{} using gjson.
- Changes made since benchmarks showed 30-40% improvement in speed.
- Follow up comments in issue https://github.com/minio/minio/issues/2208
for more details.
- gjson parsing of parts from xl.json for listParts.
- gjson parsing of statInfo from xl.json for getObjectInfo.
- Vendorizing gjson dependency.
master
Karthic Rao
8 years ago
committed by
Harshavardhana
parent
66459a4ce0
commit
8bd78fbdfb
@ -0,0 +1,20 @@ |
||||
The MIT License (MIT) |
||||
|
||||
Copyright (c) 2016 Josh Baker |
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of |
||||
this software and associated documentation files (the "Software"), to deal in |
||||
the Software without restriction, including without limitation the rights to |
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of |
||||
the Software, and to permit persons to whom the Software is furnished to do so, |
||||
subject to the following conditions: |
||||
|
||||
The above copyright notice and this permission notice shall be included in all |
||||
copies or substantial portions of the Software. |
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS |
||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR |
||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER |
||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN |
||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
||||
@ -0,0 +1,278 @@ |
||||
<p align="center"> |
||||
<img |
||||
src="logo.png" |
||||
width="240" height="78" border="0" alt="GJSON"> |
||||
<br> |
||||
<a href="https://travis-ci.org/tidwall/gjson"><img src="https://img.shields.io/travis/tidwall/gjson.svg?style=flat-square" alt="Build Status"></a><!-- |
||||
<a href="http://gocover.io/github.com/tidwall/gjson"><img src="https://img.shields.io/badge/coverage-97%25-brightgreen.svg?style=flat-square" alt="Code Coverage"></a> |
||||
--> |
||||
<a href="https://godoc.org/github.com/tidwall/gjson"><img src="https://img.shields.io/badge/api-reference-blue.svg?style=flat-square" alt="GoDoc"></a> |
||||
</p> |
||||
|
||||
<p align="center">get a json value quickly</a></p> |
||||
|
||||
GJSON is a Go package the provides a [very fast](#performance) and simple way to get a value from a json document. The reason for this library it to give efficient json indexing for the [BuntDB](https://github.com/tidwall/buntdb) project. |
||||
|
||||
Getting Started |
||||
=============== |
||||
|
||||
## Installing |
||||
|
||||
To start using GJSON, install Go and run `go get`: |
||||
|
||||
```sh |
||||
$ go get -u github.com/tidwall/gjson |
||||
``` |
||||
|
||||
This will retrieve the library. |
||||
|
||||
## Get a value |
||||
Get searches json for the specified path. A path is in dot syntax, such as "name.last" or "age". This function expects that the json is well-formed and validates. Invalid json will not panic, but it may return back unexpected results. When the value is found it's returned immediately. |
||||
|
||||
```go |
||||
package main |
||||
|
||||
import "github.com/tidwall/gjson" |
||||
|
||||
const json = `{"name":{"first":"Janet","last":"Prichard"},"age":47}` |
||||
|
||||
func main() { |
||||
value := gjson.Get(json, "name.last") |
||||
println(value.String()) |
||||
} |
||||
``` |
||||
|
||||
This will print: |
||||
|
||||
``` |
||||
Prichard |
||||
``` |
||||
|
||||
## Path Syntax |
||||
|
||||
A path is a series of keys separated by a dot. |
||||
A key may contain special wildcard characters '\*' and '?'. |
||||
To access an array value use the index as the key. |
||||
To get the number of elements in an array or to access a child path, use the '#' character. |
||||
The dot and wildcard characters can be escaped with '\'. |
||||
|
||||
```json |
||||
{ |
||||
"name": {"first": "Tom", "last": "Anderson"}, |
||||
"age":37, |
||||
"children": ["Sara","Alex","Jack"], |
||||
"fav.movie": "Deer Hunter", |
||||
"friends": [ |
||||
{"first": "James", "last": "Murphy"}, |
||||
{"first": "Roger", "last": "Craig"} |
||||
] |
||||
} |
||||
``` |
||||
``` |
||||
"name.last" >> "Anderson" |
||||
"age" >> 37 |
||||
"children.#" >> 3 |
||||
"children.1" >> "Alex" |
||||
"child*.2" >> "Jack" |
||||
"c?ildren.0" >> "Sara" |
||||
"fav\.movie" >> "Deer Hunter" |
||||
"friends.#.first" >> [ "James", "Roger" ] |
||||
"friends.1.last" >> "Craig" |
||||
``` |
||||
To query an array: |
||||
``` |
||||
`friends.#[last="Murphy"].first` >> "James" |
||||
``` |
||||
|
||||
## Result Type |
||||
|
||||
GJSON supports the json types `string`, `number`, `bool`, and `null`. |
||||
Arrays and Objects are returned as their raw json types. |
||||
|
||||
The `Result` type holds one of these: |
||||
|
||||
``` |
||||
bool, for JSON booleans |
||||
float64, for JSON numbers |
||||
string, for JSON string literals |
||||
nil, for JSON null |
||||
``` |
||||
|
||||
To directly access the value: |
||||
|
||||
```go |
||||
result.Type // can be String, Number, True, False, Null, or JSON |
||||
result.Str // holds the string |
||||
result.Num // holds the float64 number |
||||
result.Raw // holds the raw json |
||||
result.Multi // holds nested array values |
||||
``` |
||||
|
||||
There are a variety of handy functions that work on a result: |
||||
|
||||
```go |
||||
result.Value() interface{} |
||||
result.Int() int64 |
||||
result.Float() float64 |
||||
result.String() string |
||||
result.Bool() bool |
||||
result.Array() []gjson.Result |
||||
result.Map() map[string]gjson.Result |
||||
result.Get(path string) Result |
||||
``` |
||||
|
||||
The `result.Value()` function returns an `interface{}` which requires type assertion and is one of the following Go types: |
||||
|
||||
```go |
||||
boolean >> bool |
||||
number >> float64 |
||||
string >> string |
||||
null >> nil |
||||
array >> []interface{} |
||||
object >> map[string]interface{} |
||||
``` |
||||
|
||||
## Get nested array values |
||||
|
||||
Suppose you want all the last names from the following json: |
||||
|
||||
```json |
||||
{ |
||||
"programmers": [ |
||||
{ |
||||
"firstName": "Janet", |
||||
"lastName": "McLaughlin", |
||||
}, { |
||||
"firstName": "Elliotte", |
||||
"lastName": "Hunter", |
||||
}, { |
||||
"firstName": "Jason", |
||||
"lastName": "Harold", |
||||
} |
||||
] |
||||
}` |
||||
``` |
||||
|
||||
You would use the path "programmers.#.lastName" like such: |
||||
|
||||
```go |
||||
result := gjson.Get(json, "programmers.#.lastName") |
||||
for _,name := range result.Array() { |
||||
println(name.String()) |
||||
} |
||||
``` |
||||
|
||||
You can also query an object inside an array: |
||||
|
||||
```go |
||||
name := gjson.Get(json, `programmers.#[lastName="Hunter"].firstName`) |
||||
println(name.String()) // prints "Elliotte" |
||||
``` |
||||
|
||||
|
||||
## Simple Parse and Get |
||||
|
||||
There's a `Parse(json)` function that will do a simple parse, and `result.Get(path)` that will search a result. |
||||
|
||||
For example, all of these will return the same result: |
||||
|
||||
```go |
||||
gjson.Parse(json).Get("name").Get("last") |
||||
gjson.Get(json, "name").Get("last") |
||||
gjson.Get(json, "name.last") |
||||
``` |
||||
|
||||
## Check for the existence of a value |
||||
|
||||
Sometimes you just want to know you if a value exists. |
||||
|
||||
```go |
||||
value := gjson.Get(json, "name.last") |
||||
if !value.Exists() { |
||||
println("no last name") |
||||
} else { |
||||
println(value.String()) |
||||
} |
||||
|
||||
// Or as one step |
||||
if gjson.Get(json, "name.last").Exists(){ |
||||
println("has a last name") |
||||
} |
||||
``` |
||||
|
||||
## Unmarshal to a map |
||||
|
||||
To unmarshal to a `map[string]interface{}`: |
||||
|
||||
```go |
||||
m, ok := gjson.Parse(json).Value().(map[string]interface{}) |
||||
if !ok{ |
||||
// not a map |
||||
} |
||||
``` |
||||
|
||||
## Performance |
||||
|
||||
Benchmarks of GJSON alongside [encoding/json](https://golang.org/pkg/encoding/json/), |
||||
[ffjson](https://github.com/pquerna/ffjson), |
||||
[EasyJSON](https://github.com/mailru/easyjson), |
||||
and [jsonparser](https://github.com/buger/jsonparser) |
||||
|
||||
``` |
||||
BenchmarkGJSONGet-8 15000000 333 ns/op 0 B/op 0 allocs/op |
||||
BenchmarkGJSONUnmarshalMap-8 900000 4188 ns/op 1920 B/op 26 allocs/op |
||||
BenchmarkJSONUnmarshalMap-8 600000 8908 ns/op 3048 B/op 69 allocs/op |
||||
BenchmarkJSONUnmarshalStruct-8 600000 9026 ns/op 1832 B/op 69 allocs/op |
||||
BenchmarkJSONDecoder-8 300000 14339 ns/op 4224 B/op 184 allocs/op |
||||
BenchmarkFFJSONLexer-8 1500000 3156 ns/op 896 B/op 8 allocs/op |
||||
BenchmarkEasyJSONLexer-8 3000000 938 ns/op 613 B/op 6 allocs/op |
||||
BenchmarkJSONParserGet-8 3000000 442 ns/op 21 B/op 0 allocs/op |
||||
``` |
||||
|
||||
JSON document used: |
||||
|
||||
```json |
||||
{ |
||||
"widget": { |
||||
"debug": "on", |
||||
"window": { |
||||
"title": "Sample Konfabulator Widget", |
||||
"name": "main_window", |
||||
"width": 500, |
||||
"height": 500 |
||||
}, |
||||
"image": { |
||||
"src": "Images/Sun.png", |
||||
"hOffset": 250, |
||||
"vOffset": 250, |
||||
"alignment": "center" |
||||
}, |
||||
"text": { |
||||
"data": "Click Here", |
||||
"size": 36, |
||||
"style": "bold", |
||||
"vOffset": 100, |
||||
"alignment": "center", |
||||
"onMouseUp": "sun1.opacity = (sun1.opacity / 100) * 90;" |
||||
} |
||||
} |
||||
} |
||||
``` |
||||
|
||||
Each operation was rotated though one of the following search paths: |
||||
|
||||
``` |
||||
widget.window.name |
||||
widget.image.hOffset |
||||
widget.text.onMouseUp |
||||
``` |
||||
|
||||
|
||||
*These benchmarks were run on a MacBook Pro 15" 2.8 GHz Intel Core i7 using Go 1.7.* |
||||
|
||||
## Contact |
||||
Josh Baker [@tidwall](http://twitter.com/tidwall) |
||||
|
||||
## License |
||||
|
||||
GJSON source code is available under the MIT [License](/LICENSE). |
||||
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 16 KiB |
@ -0,0 +1,20 @@ |
||||
The MIT License (MIT) |
||||
|
||||
Copyright (c) 2016 Josh Baker |
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of |
||||
this software and associated documentation files (the "Software"), to deal in |
||||
the Software without restriction, including without limitation the rights to |
||||
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of |
||||
the Software, and to permit persons to whom the Software is furnished to do so, |
||||
subject to the following conditions: |
||||
|
||||
The above copyright notice and this permission notice shall be included in all |
||||
copies or substantial portions of the Software. |
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS |
||||
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR |
||||
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER |
||||
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN |
||||
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
||||
@ -0,0 +1,31 @@ |
||||
Match |
||||
===== |
||||
<a href="https://travis-ci.org/tidwall/match"><img src="https://img.shields.io/travis/tidwall/match.svg?style=flat-square" alt="Build Status"></a> |
||||
<a href="https://godoc.org/github.com/tidwall/match"><img src="https://img.shields.io/badge/api-reference-blue.svg?style=flat-square" alt="GoDoc"></a> |
||||
|
||||
Match is a very simple pattern matcher where '*' matches on any |
||||
number characters and '?' matches on any one character. |
||||
Installing |
||||
---------- |
||||
|
||||
``` |
||||
go get -u github.com/tidwall/match |
||||
``` |
||||
|
||||
Example |
||||
------- |
||||
|
||||
```go |
||||
match.Match("hello", "*llo") |
||||
match.Match("jello", "?ello") |
||||
match.Match("hello", "h*o") |
||||
``` |
||||
|
||||
|
||||
Contact |
||||
------- |
||||
Josh Baker [@tidwall](http://twitter.com/tidwall) |
||||
|
||||
License |
||||
------- |
||||
Redcon source code is available under the MIT [License](/LICENSE). |
||||
@ -0,0 +1,192 @@ |
||||
// Match provides a simple pattern matcher with unicode support.
|
||||
package match |
||||
|
||||
import "unicode/utf8" |
||||
|
||||
// Match returns true if str matches pattern. This is a very
|
||||
// simple wildcard match where '*' matches on any number characters
|
||||
// and '?' matches on any one character.
|
||||
|
||||
// pattern:
|
||||
// { term }
|
||||
// term:
|
||||
// '*' matches any sequence of non-Separator characters
|
||||
// '?' matches any single non-Separator character
|
||||
// c matches character c (c != '*', '?', '\\')
|
||||
// '\\' c matches character c
|
||||
//
|
||||
func Match(str, pattern string) bool { |
||||
if pattern == "*" { |
||||
return true |
||||
} |
||||
return deepMatch(str, pattern) |
||||
} |
||||
func deepMatch(str, pattern string) bool { |
||||
for len(pattern) > 0 { |
||||
if pattern[0] > 0x7f { |
||||
return deepMatchRune(str, pattern) |
||||
} |
||||
switch pattern[0] { |
||||
default: |
||||
if len(str) == 0 { |
||||
return false |
||||
} |
||||
if str[0] > 0x7f { |
||||
return deepMatchRune(str, pattern) |
||||
} |
||||
if str[0] != pattern[0] { |
||||
return false |
||||
} |
||||
case '?': |
||||
if len(str) == 0 { |
||||
return false |
||||
} |
||||
case '*': |
||||
return deepMatch(str, pattern[1:]) || |
||||
(len(str) > 0 && deepMatch(str[1:], pattern)) |
||||
} |
||||
str = str[1:] |
||||
pattern = pattern[1:] |
||||
} |
||||
return len(str) == 0 && len(pattern) == 0 |
||||
} |
||||
|
||||
func deepMatchRune(str, pattern string) bool { |
||||
var sr, pr rune |
||||
var srsz, prsz int |
||||
|
||||
// read the first rune ahead of time
|
||||
if len(str) > 0 { |
||||
if str[0] > 0x7f { |
||||
sr, srsz = utf8.DecodeRuneInString(str) |
||||
} else { |
||||
sr, srsz = rune(str[0]), 1 |
||||
} |
||||
} else { |
||||
sr, srsz = utf8.RuneError, 0 |
||||
} |
||||
if len(pattern) > 0 { |
||||
if pattern[0] > 0x7f { |
||||
pr, prsz = utf8.DecodeRuneInString(pattern) |
||||
} else { |
||||
pr, prsz = rune(pattern[0]), 1 |
||||
} |
||||
} else { |
||||
pr, prsz = utf8.RuneError, 0 |
||||
} |
||||
// done reading
|
||||
for pr != utf8.RuneError { |
||||
switch pr { |
||||
default: |
||||
if srsz == utf8.RuneError { |
||||
return false |
||||
} |
||||
if sr != pr { |
||||
return false |
||||
} |
||||
case '?': |
||||
if srsz == utf8.RuneError { |
||||
return false |
||||
} |
||||
case '*': |
||||
return deepMatchRune(str, pattern[prsz:]) || |
||||
(srsz > 0 && deepMatchRune(str[srsz:], pattern)) |
||||
} |
||||
str = str[srsz:] |
||||
pattern = pattern[prsz:] |
||||
// read the next runes
|
||||
if len(str) > 0 { |
||||
if str[0] > 0x7f { |
||||
sr, srsz = utf8.DecodeRuneInString(str) |
||||
} else { |
||||
sr, srsz = rune(str[0]), 1 |
||||
} |
||||
} else { |
||||
sr, srsz = utf8.RuneError, 0 |
||||
} |
||||
if len(pattern) > 0 { |
||||
if pattern[0] > 0x7f { |
||||
pr, prsz = utf8.DecodeRuneInString(pattern) |
||||
} else { |
||||
pr, prsz = rune(pattern[0]), 1 |
||||
} |
||||
} else { |
||||
pr, prsz = utf8.RuneError, 0 |
||||
} |
||||
// done reading
|
||||
} |
||||
|
||||
return srsz == 0 && prsz == 0 |
||||
} |
||||
|
||||
var maxRuneBytes = func() []byte { |
||||
b := make([]byte, 4) |
||||
if utf8.EncodeRune(b, '\U0010FFFF') != 4 { |
||||
panic("invalid rune encoding") |
||||
} |
||||
return b |
||||
}() |
||||
|
||||
// Allowable parses the pattern and determines the minimum and maximum allowable
|
||||
// values that the pattern can represent.
|
||||
// When the max cannot be determined, 'true' will be returned
|
||||
// for infinite.
|
||||
func Allowable(pattern string) (min, max string) { |
||||
if pattern == "" || pattern[0] == '*' { |
||||
return "", "" |
||||
} |
||||
|
||||
minb := make([]byte, 0, len(pattern)) |
||||
maxb := make([]byte, 0, len(pattern)) |
||||
var wild bool |
||||
for i := 0; i < len(pattern); i++ { |
||||
if pattern[i] == '*' { |
||||
wild = true |
||||
break |
||||
} |
||||
if pattern[i] == '?' { |
||||
minb = append(minb, 0) |
||||
maxb = append(maxb, maxRuneBytes...) |
||||
} else { |
||||
minb = append(minb, pattern[i]) |
||||
maxb = append(maxb, pattern[i]) |
||||
} |
||||
} |
||||
if wild { |
||||
r, n := utf8.DecodeLastRune(maxb) |
||||
if r != utf8.RuneError { |
||||
if r < utf8.MaxRune { |
||||
r++ |
||||
if r > 0x7f { |
||||
b := make([]byte, 4) |
||||
nn := utf8.EncodeRune(b, r) |
||||
maxb = append(maxb[:len(maxb)-n], b[:nn]...) |
||||
} else { |
||||
maxb = append(maxb[:len(maxb)-n], byte(r)) |
||||
} |
||||
} |
||||
} |
||||
} |
||||
return string(minb), string(maxb) |
||||
/* |
||||
return |
||||
if wild { |
||||
r, n := utf8.DecodeLastRune(maxb) |
||||
if r != utf8.RuneError { |
||||
if r < utf8.MaxRune { |
||||
infinite = true |
||||
} else { |
||||
r++ |
||||
if r > 0x7f { |
||||
b := make([]byte, 4) |
||||
nn := utf8.EncodeRune(b, r) |
||||
maxb = append(maxb[:len(maxb)-n], b[:nn]...) |
||||
} else { |
||||
maxb = append(maxb[:len(maxb)-n], byte(r)) |
||||
} |
||||
} |
||||
} |
||||
} |
||||
return string(minb), string(maxb), infinite |
||||
*/ |
||||
} |
||||
Loading…
Reference in new issue