DPIR：Data Propagation Intermediate Representation
Traditional data propagation generally use DXL(Data eXchange Language e.g. XML, YAML, JSON) as the data carrier.
This approach have two penalties:
- DXLs are very high-level, convenient for user to edit but Inconvenient for program to parse.
- There are too many DXLs exist, programs use different DXLs can't exchange data directly.
Here I introduce a better way for data propagation: design a intermediate representation for data propagation.
I name it DPIR(Data Propagating Intermediate Representation).
DPIR should be low-level enough for programs to access efficiently and expressive enough for transforming to and from DXLs with no information lose.
Preliminary Design of DPIR
String encoding is another problem for data propagation, to avoid this problem DPIR save strings in a individual file: StrFile.
Non-string data is saved in DataFile.
StrFile
StrFile's name form is: name.dpir-str-encode.
Where name is the source file's name and encode is the string encoding used.
For example: DPIR from config.json, it's StrFile's name should be config.dpir-str-encode.
For the StrFile saved with UTF-8 it's name should be config.dpir-str-utf8;
For the StrFile saved with UTF-16 LE it's name should be config.dpir-str-utf16le;
For the StrFile saved with UTF-16 BE it's name should be config.dpir-str-utf16be.
There can be multiple StrFiles's saved with different encodings in a single DPIR.
The strings in StrFile are terminated with \n and accessed with index started from 1, the index 0 is used to represent empty string("").
Therefore the line number of a line is also the index of the string saved in this line.
DataFile
DataFile's name form is: name.dpir-data where name is source file's name.
There's a format description in GO-like syntax for DataFile:
// structure of whole file, fields aligned to block size。
type full_file struct{
fileHead file_head // fileHead holds some global metadata.
root array_table/map_tabl/list_table/struct_table/string/integer/float/complex
// field root is root of the whole data structure, it can be a array table, a map table, a list table, a struct table, a string, a integer, a float-point number, or a complex number.
// followed by subsequent tables(if root is a table).
}
//--------------------------------------------------
//*******************************************************************
type file_head struct{
blockSizeLog uint8 // specified blockSize：blockSize=1 << blockSizeLog
fileHeadSize uint8 // size of file_head, block unit.
majorVersion uint8 // major version.
minorVersion uint8 // minor version.
generalEncodeFlag byte // specified UTF and ASCII encoding, can be combined with '|'.
charSetFlag byte // specified char set.
encodeFormatFlag byte // specified encodings of char set specified by charSetFlag, can be combined with '|'.
rootTypeFlag byte // specified type of root
sonKeyTypeFlag byte // (when rootTypeFlag is TYPE_MAP)specified keyType of all entrys in root table.
sonValueTypeFlag byte // (when rootTypeFlag is TYPE_ARRAY or TYPE_MAP)specified valueType of all entrys in root table.
reserveFileHead [n]byte // reserve space, used to align file_head to block size.
}
// generalEncodeFlag:
const(
ASCII = 1<<iota
UTF8
UTF16LE
UTF16BE
UTF32LE
UTF32BE
)
// charSetFlag:
const(
CHINESE = 1+iota // Chinese char set.
JAPANESE // Japanese char set。
// and so on
)
// encodeFormatFlag:
const(
FLAG1 = 1<<iota
FLAG2
FLAG3
FLAG4
FLAG5
FLAG6
FLAG7
FLAG8
)
// For example:
// When charSetFlag is CHINESE, FLAG1 represent GBK, FLAG2 represent GB2312, FLAG3 represent GB18030 ...
// When charSetFlag is JAPANESE, FLAG1 represent Shift_JIS, FLAG2 represent EUC_JP, FLAG3 represent ISO-2022-JP ...
// typeFlag: for rootTypeFlag，keyTypeFlag，valueTypeFlag，sonKeyTypeFlag and sonValueTypeFlag.
const(
TYPE_ARRAY = iota // array type
TYPE_MAP // map type
TYPE_LIST // list type
TYPE_STRUCT // struct type
TYPE_STRING // string type
TYPE_INT8 // and so on...
TYPE_INT16
TYPE_INT32
TYPE_INT64
TYPE_INT128
TYPE_INT256
TYPE_UINT8
TYPE_UINT16
TYPE_UINT32
TYPE_UINT64
TYPE_UINT128
TYPE_UINT256
TYPE_FLOAT16
TYPE_FLOAT32
TYPE_FLOAT64
TYPE_FLOAT128
TYPE_FLOAT256
TYPE_FLOAT512
TYPE_COMPLEX32
TYPE_COMPLEX64
TYPE_COMPLEX128
TYPE_COMPLEX256
TYPE_COMPLEX512
TYPE_COMPLEX1024
)
//----------------------------------------------------------------------
//*******************************************************************
// array table：
type array_table struct{
entryNum uint16 // number of entrys in current table.
sonKeyTypeFlag byte // this field only exist when value of all entrys in current table is map_table(specified by father entry's sonValueType), this field specified keyType of son table's all entrys.
sonValueTypeFlag byte // this field only exist when value of all entrys in current table is map_table or array_table, this field specified valueType of son table's all entrys.
entrys [entryNum]array_entry // entrys of current table.
reserveTable [n]byte // reserved space, used to align current table to block size.
}
// array entry:
type array_entry struct{
value [valueSize]byte
// according to father entry's sonValueTypeFlag：
// if it's TYPE_ARRAY, TYPE_MAP, TYPE_LIST or TYPE_STRUCT，then value is uint16 type，specified son table's offset, relative to current table's end, block unit.
// if it's TYPE_STRING, then value is uint16 type, specified the index of string in *StrFile*.
// if it's other type(e.g. int64, complex128), then value is the real value of this entry.
}
//---------------------------------------------------------------------------------------
//*******************************************************************************************
// map table:
type map_table struct{
entryNum uint16 // specified nuber of entrys in current table.
sonKeyTypeFlag byte // this field only exist when value of all entrys in current table is map_table(specified by father entry's sonValueType), this field specified keyType of son table's all entrys.
sonValueTypeFlag byte // this field only exist when value of all entrys in current table is map_table or array_table, this field specified valueType of son table's all entrys.
entrys [entryNum]map_entry // entrys of current table.
reserveTable [n]byte // reserved space, used to align current table to block size.
}
// map entry
type map_entry struct{
key [keySize]byte // current entry's key, type and size is specified by father entry's sonKeyTypeFlag;
// if father entry's sonKeyTypeFlag is TYPE_STRING, then key is uint16 type, specified the index of string in *StrFile*.
value [valueSize]byte // current entry's value, according to father entry's sonValueTypeFlag:
// if it's TYPE_ARRAY, TYPE_MAP, TYPE_LIST or TYPE_STRUCT，then value is uint16 type，specified son table's offset, relative to current table's end, block unit.
// if it's TYPE_STRING, then value is uint16 type, specified the index of string in *StrFile*.
// if it's other type(e.g. int64, complex128), then value is the real value of this entry.
}
//-------------------------------------------------------------------------------------
//*************************************************************************************
// list table:
type list_table struct{
entryNum uint16 // specified nuber of entrys in current table.
entrys [entryNum]list_entry // entrys in current table.
exValues [n]byte // real values for types beyond size of 32bit(e.g. int64, complex128).
reserveTable [n]byte // reserved space, used to align current table to block size.
}
// list entry:
type list_entry struct{
valueTypeFlag byte // specified valueType of current entry.
sonKeyTypeFlag byte // this field only exist when valueTypeFlag is TYPE_MAP, this field specified keyType of son table's all entrys.
sonValueTypeFlag byte // this field only exist when valueTypeFlag is TYPE_MAP or TYPE_ARRAY, this field specified valueType of son table's all entrys.
value [n]byte // the "value" of current entry.
reserveEntry [n]byte // reserved space, used to align current entry to 5 bytes.
// according to valueTypeFlag：
// if it's TYPE_ARRAY, TYPE_MAP, TYPE_LIST or TYPE_STRUCT，then value is uint16 type，specified son table's offset, relative to current table's end, block unit.
// if it's TYPE_STRING, then value is uint16 type, specified the index of string in *StrFile*.
// if it's number type below size of 32bit(int8-32, uint8-32, float32), then value is the real value of current entry.
// if it's number type beyond size of 32bit, then value is uint32 type, specified the offset of real value, relative to start of exValues, byte unit.
}
//---------------------------------------------------------------------------------------
//*******************************************************************************************
// struct table:
type struct_table struct{
entryNum byte // number of entrys in current table.
entrys [entryNum]struct_entry // entrys of current table.
exValues [n]byte // real values for types beyond size of 32bit(e.g. int64, complex128).
reserveTable [n]byte // reserved space, used to align current table to block size.
}
// struct entry：
type struct_entry struct{
valueTypeFlag byte // specified valueType of current entry.
key uint16 // index of string in *StrFile*.
sonKeyTypeFlag byte // this field only exist when valueTypeFlag is TYPE_MAP, this field specified keyType of son table's all entrys.
sonValueTypeFlag byte // this field only exist when valueTypeFlag is TYPE_MAP or TYPE_ARRAY, this field specified valueType of son table's all entrys.
value [n]byte // the "value" of current entry.
reserveEntry [n]byte // reserved space, used to align current entry to 7 bytes.
// according to valueTypeFlag：
// if it's TYPE_ARRAY, TYPE_MAP, TYPE_LIST or TYPE_STRUCT，then value is uint16 type，specified son table's offset, relative to current table's end, block unit.
// if it's TYPE_STRING, then value is uint16 type, specified the index of string in *StrFile*.
// if it's number type below size of 32bit(int8-32, uint8-32, float32), then value is the real value of current entry.
// if it's number type beyond size of 32bit, then value is uint32 type, specified the offset of real value, relative to start of exValues, byte unit.
}
Top comments (0)