Fatal problems with es's save All method, and replacing save All with updates

Recently, we encountered a problem with es on our project, because we called the start-es encapsulated Api [saveAll] method to import es into the library. Through reading the source code and repeated testing, saveAll found that it works by updating the documents in the index by id, and overwriting all the data in the original document. However, this is completely contrary to our requirements. Due to business problems, we will store the same document in the database several times, and each time some fields will be null, which is unavoidable for business requirements. Since we used saveAll in the initial design, our data will always be incomplete. Found a way to replace saveAll.
To explain this method briefly:

1. update the field when it does not exist in the corresponding document
2. If the field exists in the corresponding document and is also available in the current database, overwrite this field
3. When the document does not exist, add the operation if the above policies are met, then the data is definitely not lost

Specific implementation is as follows

/**
 * ElasticsearchUtil
 * 智慧
 * @author lhy
 * @date 2023年08月23日 11:40
 */
@Slf4j
@Component
public class ElasticsearchUtil {

    @Autowired
    private RestHighLevelClient restHighLevelClient;

    /**
     * es的批量【插入/更新】操作
     * @author lhy
     * @date 2023/8/23 11:27
     * @param esIndex es索引
     * @param paramList 批量插入的数据集，其中每一个【Map<S,O>】 都是一个文档数据,该Map应由具体的实体类进行映射填充。并且仅应存放不为null的数据
     *                  以此,上述期望将会达到目标。
     */
    public void insertWisdom(String esIndex, List<Map<String,Object>> paramList) {
        //批量请求
        BulkRequest bulkRequest = new BulkRequest();
        //遍历文档数据集
        for (Map<String, Object> param : paramList) {
            //指定索引及文档
            UpdateRequest request = new UpdateRequest(esIndex, param.get("id").toString());
            //如果文档存在，则更新文档内容为参数 param 中的内容；如果文档不存在，则插入一个新文档，内容为参数 param 中的内容。
            request.doc(param).docAsUpsert(true).upsertRequest();
            bulkRequest.add(request);
        }
        //插入用户信息
        try {
            BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, RequestOptions.DEFAULT);
            if (bulk.hasFailures()) {
                log.info("数据写入失败：{}",bulk.buildFailureMessage());
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    /**
     * 实体类数据提取
     * @author lhy
     * @date 2023/8/23 13:42
     * @param obj 实体类
     * @param claxx class
     * @return java.util.Map<java.lang.String,java.lang.Object>
     */
    private static Map<String, Object> getPureData(Object obj,Class<?> claxx) {
        Map<String, Object> map = new HashMap<>();
        Field[] fields = claxx.getDeclaredFields();
        for (Field field : fields) {
            field.setAccessible(true);
            try {
                Object value = field.get(obj);
                //过滤掉值为null的数据
                if (Objects.nonNull(value)) {
                    map.put(field.getName(), value);
                }
            } catch (IllegalAccessException e) {
                e.printStackTrace();
            }
        }
        return map;
    }

    public static Map<String,Object> getPureData(Object obj) {
        String value = JSON.toJSONString(obj);
        return JSON.parseObject(value, Map.class);
    }

}

The core idea is in the design of getPureData, where we only put data whose entity attribute value is not null

If you have good ideas and methods to share with me, the first time to use this platform, I hope it will not affect your experience

Comments

[Vincent A. Cicirello]

Hi. Welcome to DEV. You can use syntax highlighting in your code blocks with:

```Java
public class Foo {

}
```

It works for most languages. Just change Java to the relevant language name. The example above would look like:

public class Foo {

}

[Wendy Wong]

Hi hongyinglai, Welcome to the DEV Community! And thank you for publishing your first article on springboot. Please feel free to include two more tags for a wider reach of your article for your readers.

[hongyinglai]

经过自动化覆盖测试发现，当对象里面的属性为集合等特殊对象时，会出现意想不到的错误，在此将Map生成改成通过json来实现，

public static Map<String,Object> getPureData(Object obj) {
        String value = JSON.toJSONString(obj);
        return JSON.parseObject(value, Map.class);
    }